95 Commits (ab8af07c19003ff66059a93c6bb1663cb562adb6)

Author	SHA1	Message	Date
Javier Jaramago Fernández	d3a34712df	Improve function resource acquisition readability ... - Improved readability for the locked map acquisition for function 'search_rules_fast_routing_dest_hg'. - Improved function documentation and added note about usage.	3 years ago
Javier Jaramago Fernández	4b5fe5888e	Fix race condition when reloading 'Query Rules Fast Routing'	3 years ago
René Cannaò	b5677e3ba9	Adding flagOUTs in mysql_query_rules.attributes ... in JSON format	3 years ago
Javier Jaramago Fernández	43372bcfc9	Optimize locking for 'search_rules_fast_routing_dest_hg' ... - Add parameter for conditionally locking inside function. - Move potential syscalls outside the 'pthread_rwlock_rdlock' section. - Reduce number of 'sprintf' calls. - Improved PROXYSQLTEST 11/15 with data randomly exceeding function default buffer size.	3 years ago
Javier Jaramago Fernández	8331858179	Remove thread variable for 'mysql-query_rules_fast_routing_algorithm' ... - Thread variable has been replaced for new variable in 'Query_Processor'. - Computation for 'mysql_query_rules_memory' is now performed by Admin during 'load_mysql_query_rules_to_runtime' instead of lazily by the MySQL_Threads.	3 years ago
Javier Jaramago Fernández	abccb39201	Add new variable 'mysql-query_rules_fast_routing_algorithm' ... - Implemented new variable 'query_rules_fast_routing_algorithm' that allows choosing between the classic per-thread hashmaps for 'query_rules_fast_routing' searches (1), and a new unique global hashmap (2). - Improved locking time on 'QueryProcessor::rwlock' during 'load_mysql_query_rules_to_runtime'. - Changed computation of 'mysql_query_rules_memory' to be lazily updated by threads themselves while building the per-threads maps. - Fixed 'ProxySQL_Admin::stats___memory_metrics' report of the memory metrics. - Improved 'PROXYSQLTEST 17' command, for being able to micro-benchmark and check 'query_rules_fast_routing_algorithm'.	3 years ago
Javier Sánchez Parra	0da59a6620	Honor reset and copy options in get_query_digests_reset_v2()	3 years ago
Javier Sánchez Parra	2c13dfdc52	Make PROXYSLQTEST 22 and 24 return the number of row affected	3 years ago
Javier Sánchez Parra	14ba7475d4	Create stats_mysql_query_digest statements directly from the digest_umap	3 years ago
Javier Sánchez Parra	6fb18bd2ac	Move get digest text code to its own function	3 years ago
Javier Sánchez Parra	28095e3109	Add auxiliary maps in get_query_digests_reset() to improve performance	3 years ago
Javier Sánchez Parra	263d645bad	Add auxiliary maps in get_query_digests() to improve performance ... Otherwise, all queries processed by ProxySQL have to wait for get_query_digest() to finish reading the maps. With this patch, ProxySQL stores query digests in the auxiliary maps while the main maps are being used by get_query_digests(). Once get_query_digests() finish reading the main maps, it merge the auxiliary maps in to the main maps and clear the content of the auxiliary maps.	3 years ago
René Cannaò	5e68075edf	Cluster: Speed up processing of MySQL Query Rules ... Multiple changes: - Query_Processor stored the resultset of query rules loaded to runtime (previously this was only for query rules fast routing) - Admin returns the stored resultset (query rules and fast routing) when queries by Cluster - In khash replaced the hashing function from the built-in __ac_X31_hash_string to CityHash32 - When Cluster is used, it calls load_mysql_query_rules_to_runtime() passing the resultsets retrieved by the remote peer - Increased SQLite cache_size to ~50MB: this seems to be a very small optimization and probably it will be reverted - pull_mysql_query_rules_from_peer() uses transactions to write to mysql_query_rules_fast_routing - (important change) pull_mysql_query_rules_from_peer() verifies the checksum of MySQL Query ules before loading them to runtime	4 years ago
René Cannaò	43c6acb218	Merge branch 'v2.x' into v2.1.2-cluster	4 years ago
André Cabaça	977bf0f1da	Removed dead code ... As reported in #3740 there was duplicated code. Originated from this merge #739	4 years ago
René Cannaò	f7bb4d201b	Add get_current_query_rules_fast_routing_count() ... Query_Processor::get_current_query_rules_fast_routing_count() now allows to quickly execute this query is ran on Admin: SELECT COUNT(*) FROM runtime_mysql_query_rules_fast_routing	5 years ago
Javier Jaramago Fernández	622a763920	Added new column 'attributes' to 'mysql_query_rules' table	5 years ago
Javier Jaramago Fernández	66478b89ee	Added new stat 'new_req_conns_count' counting the num of 'create_new_connection' recieved	6 years ago
Javier Jaramago Fernández	4409044398	Added support for annotation 'create_new_connection' that prevents reusing backend connections	6 years ago
René Cannaò	9c2f0bf3ac	Adding tests to benchmark multiple lookups in mysql_query_rules_fast_routing	6 years ago
René Cannaò	d15f055b1b	Remove define PROXYSQL_QPRO_PTHREAD_MUTEX ... Removed also all old code that didn't use pthread mutex	6 years ago
René Cannaò	c4a0ea9d4d	Remove all code not related to FAST_ROUTING_NEW208 ... Removed also #define FAST_ROUTING_NEW208 This was new code introduced in 2.0.8 , and now is the default	6 years ago
val	a095ef00d3	use DIGEST_STATS_FAST_1 always	6 years ago
René Cannaò	c4607d94fe	class Web_Interface_plugin friends Query_Processor	6 years ago
Kir Shatrov	3b031f3518	Prevent crash on index() returning NULL ... in handler___client_DSS_QUERY_SENT___server_DSS_NOT_INITIALIZED__get_connection	6 years ago
Kir Shatrov	2f7bb1a1cf	Validate gtid input	6 years ago
Kir Shatrov	0922949b8c	Support min_gtid comment	6 years ago
René Cannaò	e9dc7a9b60	Implementation of SQLI fingerprints whitelist ... libsqlinjection generates a lot of false positives. This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints This table can list fingerprints generated by libsqlinjection: if the fingerprint is listed in this table, proxysql will consider it as a false positive. This commit also enables SQLi algorithm only if the query is not already explicitly whitelisted.	6 years ago
René Cannaò	485d313514	Further development of firewall whitelist. ... * introduced 2 new variables: - mysql-firewall_whitelist_enabled : defines if the firewall with whitelist algorithm is enabled or not - firewall_whitelist_errormsg : error message returned to the client if an error message is not already set * Query_Processor returns firewall_whitelist_mode to the calling MySQL_Session, but the logging on error log currently happens within the Query_Processor itself * if mysql-firewall_whitelist_enabled is turned on and a user it is not found in mysql_firewall_whitelist_users , the default behavior is that query is rejected * lookup on mysql_firewall_whitelist_users happens for username@ip (specific IP) and username@'' (all IPs) . Match on network subnet is currently not implemented * implemented SAVE MYSQL FIREWALL FROM RUNTIME * fixed implementation of SAVE MYSQL FIREWALL TO DISK * runtime configuration can be viewed in runtime_mysql_firewall_whitelist_rules and runtime_mysql_firewall_whitelist_users	6 years ago
René Cannaò	94f27716b7	Preparing SAVE MYSQL FIREWALL FROM RUNTIME	6 years ago
René Cannaò	6ee2a0cb19	Load mysql firewall whitelist rules to runtime	6 years ago
René Cannaò	4005746b1a	More on firewall whitelist	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist ... Improved also compiling time	6 years ago
René Cannaò	6bed334db1	stats_history.history_mysql_query_digest #2368 ... Implementation of new table stats_history.history_mysql_query_digest #2368 Implemented command `SAVE MYSQL DIGEST TO DISK`. Save to disk with really minimal memory footprint compared to querying `stats_mysql_query_digest` or `stats_mysql_query_digest_reset` . Implemented new admin variable `admin-stats_mysql_query_digest_to_disk` to automatically save to disk.	6 years ago
René Cannaò	a0ea0c118c	V2.0.8 fast routing (#2309) ... * Implementation of unit tests related to mysql_query_rules_fast_routing * Implementation of rules_fast_routing with khash * Adding memory usage for new fast routing algo	6 years ago
René Cannaò	2fec51c0a9	A lot of changes to make query digest map faster ... Extended SQLite3_result to support a mutex for concurrent rows insert. Extensively rewritten get_query_digests() and get_query_digests_reset() to support multiple threads, and concurrently generate a resultset. Function get_query_digests_reset() defer the deletion of objects from query digest map after generating the resultset for Admin. This drastically reduce the amount of time that query digest map is locked. Rewritten get_query_digests_total_size() to make it multi-threaded. Added 3 new functions for purging: - purge_query_digests() : wrapper - purge_query_digests_sync() : synchronous purge of query digest map, either single-threaded or multi-threaded - purge_query_digests_async() : asynchronous purge of query digest map, single-threaded only If table `stats_mysql_query_digest_reset` is queried but not `stats_mysql_query_digest`, only the first one is populated. If table `stats_mysql_query_digest_reset` is queried and also `stats_mysql_query_digest`, the second one is populated as a copy from the first. Implemented new SQL `TRUNCATE [TABLE]` for `[stats.]stats_mysql_query_digest[_reset]`: this will purge the query digest map as fast as possible without creating any table in SQLite. For testing/benchmark: - modified ProxySQL_Test___GenerateRandomQueryInDigestTable() to generate more random data and randomly switch `mysql_thread___query_digests_normalize_digest_text` Extended command `PROXYSQLTEST` with the following tests: - 1 : generates entries in digest map - 2 : gets all the entries from the digest map, but without writing to DB, with multiple threads - 3 : gets all the entries from the digest map and reset, but without writing to DB, with multiple threads - 4 : purges the digest map, synchronously, in single thread - 5 : purges the digest map, synchronously, in multiple threads - 6 : purges the digest map, asynchronously, in single thread (this seems the faster) Replaced some sprintf() with new function my_itoa() . Modified class QP_query_digest_stats() to include username, schemaname and client_address if no longer than 24 bytes. This makes storage requirement for QP_query_digest_stats() bigger, but reduces a lot of calls to memory management. Modified QP_query_digest_stats::get_row() to drastically reduce the number of malloc/free and string copy. Overall, this commit has the following effects: - it increases the memory usage for query digest map. On test workload, this results in 10% more memory usage - drastically reduces the calls to memory management - populating `stats_mysql_query_digest_reset` is 3 times faster - the amount of time in which query digest map is locked is reduced by 95%	7 years ago
René Cannaò	6aac86bebc	Changed the way max_lag_ms in query is computed ... It now remembers if the query is waiting for long time	7 years ago
René Cannaò	af0a7100e2	Adding support for max_lag_ms in query	7 years ago
René Cannaò	9b14103236	Collect memory statistics about query rules	7 years ago
René Cannaò	0a60cfda11	Merge branch 'v2.0.0-tb' into v2.0.0-merge-tb	7 years ago
René Cannaò	6f8c37eec9	Store digest_text on a different hash table ... This should drastically reduce memory usage where identical queries are executed an several schemas or by different users.	7 years ago
René Cannaò	c5cff7a921	Added client_address in stats_mysql_query_digest	7 years ago
René Cannaò	72724b22e0	Preparing for support of cache_timeout ... Adding column mysql_query_rules.cache_timeout . Functionality is not implemented yet, only the column is added.	7 years ago
René Cannaò	f666d40db2	Do not cache empty resultset, rule setting #1696	8 years ago
René Cannaò	c15ee49797	Add support for wildcard in mysql_query_rules.client_addr #1450 ... Input validation: - client_addr not longer than INET6_ADDRSTRLEN - % allowed only at the end of client_addr Query rule itself remembers if there is a wildcard or not. If there is a wildcard compares string till the wildcard. If client_addr=='%' , it is a match all.	8 years ago
René Cannaò	b5d8c4d3ba	Added table mysql_query_rules_fast_routing ... This table is responsible for performing routing for very specific use case scenarios, like having tens of thousands of schemas	8 years ago
Nikolaos Vyzas	9f3c265fd9	Resolving conflicts for "Add support for GTID consistent reads"	8 years ago
Nick Vyzas	bfcf57e9d0	Updated 'hits' to uint64 to prevent overflow	8 years ago
René Cannaò	1414d4430f	Introduced new table stats_memory_metrics ... Here an example: ``` Admin> SELECT * FROM stats_memory_metrics ORDER BY variable_name; +----------------------+----------------+ | Variable_Name | Variable_Value | +----------------------+----------------+ | Auth_memory | 836 | | SQLite3_memory_bytes | 1636648 | | jemalloc_active | 3469312 | | jemalloc_allocated | 3055248 | | jemalloc_mapped | 14680064 | | jemalloc_metadata | 1480576 | | jemalloc_resident | 4542464 | | jemalloc_retained | 0 | | query_digest_memory | 459 | +----------------------+----------------+ 9 rows in set (0.00 sec) ```	9 years ago