New command `PROXYSQL FLUSH CONFIGDB` allows to close and reopen configdb (`proxysql.db` on disk).
Note:
It is important that the (new) configdb is a valid one.
A misconfigured configdb will cause undefined behaviors.
The 2 new variables are:
* mysql-query_digests_max_digest_length : defines the maximum length of digest_text as reported in stats_mysql_query_digest
* mysql-query_digests_max_query_length : defines the maximum query length processed when computing query's digest and digext_text
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
This commit is able to speed up LOAD MYSQL SERVERS TO RUNTIME up to 10x for very large mysql_servers table with tens of thousands of servers.
Main improvements:
* removed 1 unnecessary call to `generate_mysql_servers_table()`
* replaced 1 call to `generate_mysql_servers_table()` with UPDATE statements when necessary
* implemented new function `servers_add()` that replaces `server_add()` to perform bulk operation
* replaced calls to `sqlite3_exec()` with proper parameters bindings
* use of multirows INSERT statements (32 rows at the time)
Miror improvements:
* added new variable mysql-hostgroup_manager_verbose , 1 by default for backward compatibility and debugging
* in not debug mode, access to HGM db has no shared cache
* in not debug mode, tables in HGM have checks disabled
Further note:
* Once issue #797 is completed, `mysql_servers` in MyHGM can be simplified further
These improvements avoid a race condition in which a prepared statement is removed from the cache before it is marked as used by the client.
This race condition was noticed when running benchmark creating 7000 _unique_ prepared statements per second.
Monitor threads are started:
* without arena cache
* with 64KB stack
malloc_conf changed from:
xmalloc:true,lg_tcache_max:16,purge:decay
to:
xmalloc:true,lg_chunk:18,lg_tcache_max:12,purge:ratio
Conflicts:
lib/MySQL_Monitor.cpp
Depending from the query, an error "Query execution was interrupted" can be returned either as:
* a standalone ERR Packet (previously handled)
* an ERR Packet at the end of a result set : this commit handle this case
Metrics introduced:
* ConnPool_get_conn_failure : connection pool cannot provide any connection
* ConnPool_get_conn_immediate : connection is provided from per-thread cache
* ConnPool_get_conn_success : the session is able to get a connection, either from per-thread cache or connection pool
MySQL_STMTs_meta() is aware to which session it belongs
entries are removed from MySQL_STMTs_meta() when unused
find_prepared_statement_by_hash() increases the client ref count
Hardcoded a maximum number of statements
The called of MySQL_STMTs_local functions doesn't must specify if it is a client or a backend.
Added also a function in MySQL_Connection to make the connection aware it is a client.
New variable admin-hash_passwords , default true
When set, passwords in mysql_users are automatically hashed in runtime
when running LOAD MYSQL USERS TO RUNTIME.
For backward compatibility, passwords in mysql_users in memory and disk aren't
automatically hashed. Although, they can be easily hashed running
SAVE MYSQL USERS TO MEMORY (and eventually SAVE MYSQL USERS TO DISK)
immediately after running LOAD MYSQL USERS TO RUNTIME
If MHM_PTHREAD_MUTEX is defined MySQL_HostGroups_Manager.h ,
MySQL_HostGroups_Manager() is compiled using pthread mutex instead od spin loop.
As a lot of computation is protected by mutex, maybe it is better not to use spin loop.
Command "LOAD SCHEDULER FROM CONFIG" is now implemented.
Config file can also be read with --initial, --reload, and when a db file is not present yet.
An example was added in src/proxysql.cfg
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
When query rules are processed, even if flagOUT is set rules are processed
in numerical order.
This new variable allows to jump back (reset rule_id to 0) N amount of times.
Using this variable is possible to implement loops.
Futhermore, a loop is possible with only one rule setting flagIN=flagOUT .
The default is 0. The maximum value is 1000000 .
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
So far, single thread sysbench is working.
Multi-threaded and resume from broken connections is still not working.
It means it still needs a lot of changes.
A new Query_Processor_Output (qpo) is allocated for every query.
Allocation is now done once per session, thus speeding up execution time
under memory profiler
This feature is not completed yet.
libmariadb/password.c is patched to use either password or sha1(password)
Class MySQL_Connection_userinfo() is expanded to also support sha1_pass
Inroduced MySQL_Authentication::set_SHA1() to store this information at runtime only
Added several auxiliary functions
This variable limits the amount of data that the client can send to the backend.
Note that ProxySQL won't really limit the amount of data that the client can send to the proxy, but the amount of data that the prox can send to the backend.
This commits also fixes issue #526Closes#526
Scheduler is being implemented as part of the Admin module.
Added table scheduler
Added table runtime_scheduler
Implemented commands:
- [LAD|SAVE] SCHEDULER [FROM|TO] [DISK|MEMORY|RUNTIME]
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
- added new function realtime_time() to get real time
- added support for multiple events, default PROXYSQL_QUERY
- MySQL_Event::write() will perfom a different action depending from log_event_type
- rewrote part of eventslog_reader_sample.cpp:
- it had few bugs
- MySQL_Event::read() will perform a different action depending from log_event_type
- time are printed in real time and not monotonic time
- all info about a query are printed in one line
These two new columns will be used to:
- enable SSL on a per server basis
- define maximum latency for a server to be consider usable: this will be a key element for multi DC support
When a server isn't responding to ping, it is flagged as shunned and all the connections need to be dropped.
All the connections must be dropped before the server is brought back online
mysql_query_rules table is changing from version to version.
In order to provide online upgrade, a new function was added:
- ProxySQL_Admin::disk_upgrade_mysql_query_rules()
Some legacy code is not currently used:
- removed reference to MySQL_Data_Stream::move_from_OUT_to_OUTpending()
- removed reference to MySQL_Data_Stream::PSarrayOUTpending
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules
Add new variable MySQL_Session::mirror (false by default)
Add new struct MySQL_Session::mirrorPkt (false by default)
These functions return immediately if this is a mirror session:
* MySQL_Protocol::generate_pkt_EOF()
* MySQL_Protocol::generate_pkt_ERR()
* MySQL_Protocol::generate_pkt_OK()
* MySQL_Protocol::generate_pkt_column_count()
* MySQL_Protocol::generate_pkt_field()
* MySQL_Protocol::generate_pkt_row3()
MySQL_ResultSet::MySQL_ResultSet() exits almost immediately if this is a mirror session
Most of the code of these functions is not executed if this is a mirror session:
* MySQL_ResultSet::add_eof()
* MySQL_ResultSet::get_resultset()
In MySQL_Session::writeout() :
* `client_myds->write_to_net_poll()` is called only if this is not a mirror session
In MySQL_Session::handler():
* if query is a `SELECT` (hardcoded for now) :
** create a new session and mark it as `mirror=true`
** duplicate the query and send it to the new session
* if `mirror==true` :
** pretend to read a new query from c`lient_myds`
** proceed normally (with the only exception that no data can be sent to a client)
In MySQL_Thread::process_all_sessions() :
* if `mirror==true` and `status==WAITING_CLIENT_DATA` it means the session is "completed" so it get destroyed
Limitations:
* it currently doesn't support large packets
* doesn't support different charset
* doesn't support transactions
All save commands now make sure that the configs are first copied from
runtime to memory before theyre read and that they are copied from
memory to runtime after they are written
Related to bug https://mariadb.atlassian.net/browse/MDEV-8338 .
Created new variable: mysql-bug_mdev_8338
If this variable is set, multiplexing is disabled if a query has NOW() or CURDATE()
Moved some code from MySQL_Session:handler_special_queries() into two new functions:
- MySQL_Session::handler_CommitRollback()
- MySQL_Session::handler_SetAutocommit()
These two new functions are called nowhere: de facto they are disabled
Features added:
* added new global variable admin-read_only: it is possible to enable read_only changing this variable and running LOAD ADMIN VARIABLES TO RUNTIME
* admin interface now replies to SHOW GLOBAL VARIABLES LIKE 'read_only'
* added new command PROXYSQL READONLY to enable read only mode
* added new command PROXYSQL READWRITE to disable read only mode : this is the only way to disable read only mode while global_variables is not readable
Note that this doesn't prevent the execution of commands LOAD and SAVE : these commands need to be used with caution
In order to understand the performance improvement related to how ProxySQL filters autocommit/commit/rollback, the follows variables were added in GLOBAL MYSQL STATUS
- Com_autocommit
- Com_autocommit_filtered
- Com_commit
- Com_commit_filtered
- Com_rollback
- Com_rollback_filtered
Summary:
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Query Cache enhancement on user/schema
Query Cache now supports distinct entries based on user/schema .
This is achieved passing to Query_Cache::set() and Query_Cache::get() the hash that identifies user and schema
Test Plan: None
Reviewers: rene
Differential Revision: http://phab.sysown.com/D3
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Added C++ headers file in cpp.h
bug fix in xchg_32 : gcc ignores it, clang doesn't
Replaced -rdynamic with --export-dynamic
Removed the override of env variables CC , CXX , CFLAGS , CPPFLAGS and LIBS
Fixed definition in MySQL_Monitor.cpp : gcc ignores it, clang doesn't
Added a compile example text file
- MySQL_Session::handler_special_queries() processes set autocommit from clients
- to the client is always returned the autocommit value set in the session
- set autocommit is either just acknowledged to the client, or forwarded to the backend based on the status of transactions
- unless there are query rules to handle SET AUTOCOMMIT , the statement is forwarded to the first found active transaction: this assumes there is just one transactions. Although multiple transactions are possible, handling multiple transactions at the same time seems a dangerous adventure
Added MySQL_Session::RequestEnd() to wrap CurrentQuery.end() and other state changes : this to minimize bugs
MySQL_Session::RequestEnd() sets query end time instead of Query_Info::end()
MySQL_Session::RequestEnd() is called also in case of special queries handled by handler_special_queries()
class MySQL_HostGroups_Manager() has now also access to admindb
added function MySQL_HostGroups_Manager::purge_mysql_servers_table() to remove entries from mysql_servers table from within MyHGM
purge_mysql_servers_table() and generate_mysql_servers_table() are called in multiple part of the code to get consistent view of mysql backends
added function MySQL_HostGroups_Manager::read_only_action(char*,int,int) responsible for triggering a traffic switch
Added 2 new functions to answer some of queries from mysqldump:
ProxySQL_Admin::generate_show_fields_from()
ProxySQL_Admin::generate_show_table_status()
ProxySQL_Admin() replies with an OK or an artificial resultset to most of the queries from mysqldump