Monitor threads are started:
* without arena cache
* with 64KB stack
malloc_conf changed from:
xmalloc:true,lg_tcache_max:16,purge:decay
to:
xmalloc:true,lg_chunk:18,lg_tcache_max:12,purge:ratio
Conflicts:
lib/MySQL_Monitor.cpp
Depending from the query, an error "Query execution was interrupted" can be returned either as:
* a standalone ERR Packet (previously handled)
* an ERR Packet at the end of a result set : this commit handle this case
Metrics introduced:
* ConnPool_get_conn_failure : connection pool cannot provide any connection
* ConnPool_get_conn_immediate : connection is provided from per-thread cache
* ConnPool_get_conn_success : the session is able to get a connection, either from per-thread cache or connection pool
MySQL_STMTs_meta() is aware to which session it belongs
entries are removed from MySQL_STMTs_meta() when unused
find_prepared_statement_by_hash() increases the client ref count
Hardcoded a maximum number of statements
The called of MySQL_STMTs_local functions doesn't must specify if it is a client or a backend.
Added also a function in MySQL_Connection to make the connection aware it is a client.
New variable admin-hash_passwords , default true
When set, passwords in mysql_users are automatically hashed in runtime
when running LOAD MYSQL USERS TO RUNTIME.
For backward compatibility, passwords in mysql_users in memory and disk aren't
automatically hashed. Although, they can be easily hashed running
SAVE MYSQL USERS TO MEMORY (and eventually SAVE MYSQL USERS TO DISK)
immediately after running LOAD MYSQL USERS TO RUNTIME
If MHM_PTHREAD_MUTEX is defined MySQL_HostGroups_Manager.h ,
MySQL_HostGroups_Manager() is compiled using pthread mutex instead od spin loop.
As a lot of computation is protected by mutex, maybe it is better not to use spin loop.
Command "LOAD SCHEDULER FROM CONFIG" is now implemented.
Config file can also be read with --initial, --reload, and when a db file is not present yet.
An example was added in src/proxysql.cfg
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
When query rules are processed, even if flagOUT is set rules are processed
in numerical order.
This new variable allows to jump back (reset rule_id to 0) N amount of times.
Using this variable is possible to implement loops.
Futhermore, a loop is possible with only one rule setting flagIN=flagOUT .
The default is 0. The maximum value is 1000000 .
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
So far, single thread sysbench is working.
Multi-threaded and resume from broken connections is still not working.
It means it still needs a lot of changes.
A new Query_Processor_Output (qpo) is allocated for every query.
Allocation is now done once per session, thus speeding up execution time
under memory profiler
This feature is not completed yet.
libmariadb/password.c is patched to use either password or sha1(password)
Class MySQL_Connection_userinfo() is expanded to also support sha1_pass
Inroduced MySQL_Authentication::set_SHA1() to store this information at runtime only
Added several auxiliary functions
This variable limits the amount of data that the client can send to the backend.
Note that ProxySQL won't really limit the amount of data that the client can send to the proxy, but the amount of data that the prox can send to the backend.
This commits also fixes issue #526Closes#526
Scheduler is being implemented as part of the Admin module.
Added table scheduler
Added table runtime_scheduler
Implemented commands:
- [LAD|SAVE] SCHEDULER [FROM|TO] [DISK|MEMORY|RUNTIME]
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
- added new function realtime_time() to get real time
- added support for multiple events, default PROXYSQL_QUERY
- MySQL_Event::write() will perfom a different action depending from log_event_type
- rewrote part of eventslog_reader_sample.cpp:
- it had few bugs
- MySQL_Event::read() will perform a different action depending from log_event_type
- time are printed in real time and not monotonic time
- all info about a query are printed in one line
These two new columns will be used to:
- enable SSL on a per server basis
- define maximum latency for a server to be consider usable: this will be a key element for multi DC support
When a server isn't responding to ping, it is flagged as shunned and all the connections need to be dropped.
All the connections must be dropped before the server is brought back online
mysql_query_rules table is changing from version to version.
In order to provide online upgrade, a new function was added:
- ProxySQL_Admin::disk_upgrade_mysql_query_rules()
Some legacy code is not currently used:
- removed reference to MySQL_Data_Stream::move_from_OUT_to_OUTpending()
- removed reference to MySQL_Data_Stream::PSarrayOUTpending
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules
Add new variable MySQL_Session::mirror (false by default)
Add new struct MySQL_Session::mirrorPkt (false by default)
These functions return immediately if this is a mirror session:
* MySQL_Protocol::generate_pkt_EOF()
* MySQL_Protocol::generate_pkt_ERR()
* MySQL_Protocol::generate_pkt_OK()
* MySQL_Protocol::generate_pkt_column_count()
* MySQL_Protocol::generate_pkt_field()
* MySQL_Protocol::generate_pkt_row3()
MySQL_ResultSet::MySQL_ResultSet() exits almost immediately if this is a mirror session
Most of the code of these functions is not executed if this is a mirror session:
* MySQL_ResultSet::add_eof()
* MySQL_ResultSet::get_resultset()
In MySQL_Session::writeout() :
* `client_myds->write_to_net_poll()` is called only if this is not a mirror session
In MySQL_Session::handler():
* if query is a `SELECT` (hardcoded for now) :
** create a new session and mark it as `mirror=true`
** duplicate the query and send it to the new session
* if `mirror==true` :
** pretend to read a new query from c`lient_myds`
** proceed normally (with the only exception that no data can be sent to a client)
In MySQL_Thread::process_all_sessions() :
* if `mirror==true` and `status==WAITING_CLIENT_DATA` it means the session is "completed" so it get destroyed
Limitations:
* it currently doesn't support large packets
* doesn't support different charset
* doesn't support transactions
All save commands now make sure that the configs are first copied from
runtime to memory before theyre read and that they are copied from
memory to runtime after they are written
Related to bug https://mariadb.atlassian.net/browse/MDEV-8338 .
Created new variable: mysql-bug_mdev_8338
If this variable is set, multiplexing is disabled if a query has NOW() or CURDATE()
Moved some code from MySQL_Session:handler_special_queries() into two new functions:
- MySQL_Session::handler_CommitRollback()
- MySQL_Session::handler_SetAutocommit()
These two new functions are called nowhere: de facto they are disabled
Features added:
* added new global variable admin-read_only: it is possible to enable read_only changing this variable and running LOAD ADMIN VARIABLES TO RUNTIME
* admin interface now replies to SHOW GLOBAL VARIABLES LIKE 'read_only'
* added new command PROXYSQL READONLY to enable read only mode
* added new command PROXYSQL READWRITE to disable read only mode : this is the only way to disable read only mode while global_variables is not readable
Note that this doesn't prevent the execution of commands LOAD and SAVE : these commands need to be used with caution
In order to understand the performance improvement related to how ProxySQL filters autocommit/commit/rollback, the follows variables were added in GLOBAL MYSQL STATUS
- Com_autocommit
- Com_autocommit_filtered
- Com_commit
- Com_commit_filtered
- Com_rollback
- Com_rollback_filtered
Summary:
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Query Cache enhancement on user/schema
Query Cache now supports distinct entries based on user/schema .
This is achieved passing to Query_Cache::set() and Query_Cache::get() the hash that identifies user and schema
Test Plan: None
Reviewers: rene
Differential Revision: http://phab.sysown.com/D3
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Added C++ headers file in cpp.h
bug fix in xchg_32 : gcc ignores it, clang doesn't
Replaced -rdynamic with --export-dynamic
Removed the override of env variables CC , CXX , CFLAGS , CPPFLAGS and LIBS
Fixed definition in MySQL_Monitor.cpp : gcc ignores it, clang doesn't
Added a compile example text file
- MySQL_Session::handler_special_queries() processes set autocommit from clients
- to the client is always returned the autocommit value set in the session
- set autocommit is either just acknowledged to the client, or forwarded to the backend based on the status of transactions
- unless there are query rules to handle SET AUTOCOMMIT , the statement is forwarded to the first found active transaction: this assumes there is just one transactions. Although multiple transactions are possible, handling multiple transactions at the same time seems a dangerous adventure
Added MySQL_Session::RequestEnd() to wrap CurrentQuery.end() and other state changes : this to minimize bugs
MySQL_Session::RequestEnd() sets query end time instead of Query_Info::end()
MySQL_Session::RequestEnd() is called also in case of special queries handled by handler_special_queries()
class MySQL_HostGroups_Manager() has now also access to admindb
added function MySQL_HostGroups_Manager::purge_mysql_servers_table() to remove entries from mysql_servers table from within MyHGM
purge_mysql_servers_table() and generate_mysql_servers_table() are called in multiple part of the code to get consistent view of mysql backends
added function MySQL_HostGroups_Manager::read_only_action(char*,int,int) responsible for triggering a traffic switch
Added 2 new functions to answer some of queries from mysqldump:
ProxySQL_Admin::generate_show_fields_from()
ProxySQL_Admin::generate_show_table_status()
ProxySQL_Admin() replies with an OK or an artificial resultset to most of the queries from mysqldump
Added status MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG as internal only (not configurable)
Added MySQL_HostGroups_Manager::replication_lag_action() to communicate replication lag to connection pool and take action
monitor_replication_lag_interval
monitor_replication_lag_timeout
Drafting MySQL_Monitor::monitor_replication_lag() as a clone of MySQL_Monitor::monitor_ping()
Status variables implemented so far:
- Client_Connections_aborted
- Client_Connections_connected
- Client_Connections_created
- Questions
- Slow_queries
Also created alias command SHOW MYSQL STATUS
- added global variables mysql-query_digests
- initialization if either mysql_thread___commands_stats or mysql_thread___query_digests are enabled
- query_parser_free() checks if QueryParserArgs was initialized
- Query_Processor::query_parser_init() checks the command type only if mysql_thread___commands_stats is true
- Query_Processor::query_parser_init() run mysql_query_digest() only if mysql_thread___query_digests is true
- Query_Processor::query_parser_update_counters() updates query digests only if digest_text is not null
- start time is always update (this is relevant for issue #327)
SET NAMES sent from the application only change the charset in the client connection.
When a backend connection is taken from the connection pool and charset mismatches, SET NAMES is sent to the backend
- Replaced some proxy_error() with daemon_log()
- glovars.proxy_restart_on_error is configurable via command line
- glovars.proxy_restart_delay is hardcoded to 1 second
- disabled listeners on telnet
- Option --admin-socket overwrite the default admin socket
signal_all_threads() now accepts an argument (default 0). If the argument is greater than 0, the threads release the thread_mutex, and sleep for the specified millisecond .
In this way processlist will immediately access all threads
If the number of free connections is greater than free_connections_pct percentage of max_connections for that specific host,
all connections idle for longer than ping_interval_server are closed instead of sending PING
- added variables mysql_thread___connect_retries_delay and mysql_thread___sessions_sort
- sessions that cannot get a connections are sorted
- commands_stats now defaults to true
Implemented LOAD MYSQL QUERY RULES FROM CONFIG via ProxySQL_Admin::Read_MySQL_Query_Rules_from_configfile()
Fixed bug in ProxySQL_Admin::save_mysql_query_rules_from_runtime()
To handle connect timeout:
- added MySQL_Data_Stream::destroy_MySQL_Connection()
- new async status: ASYNC_CONNECT_TIMEOUT
- "select @@version_comment limit 1" is answered by proxysql and not forwarded to any backend
- added MySQL_Data_Stream variables connect_retries_on_failure and max_connect_time
- added global variables mysql-connect_timeout_server_max and mysql-connect_retries_on_failure
Improvement: proxysql returns more info in COM_STATISTICS: clients connected and uptime
Simplified the ping within MySQL_Thread
SQLite3_row::add_fields() to handle empty string
prot_status was not initialized in MySQL_Protocol
Session's default schema was always mysql_thread___default_schema , now fixed with correct value
default_schema is now read from config file
Dependencies:
- added libevent
- added mariadb-connector-c for non-blocking API
- added patch for mariadb-connector-c ( see https://mariadb.atlassian.net/browse/CONC-131 )
MySQL Monitoring:
- added table mysql_server_connect_log
- added table mysql_server_ping_log
- added access to admindb
- implemented monitoring using MariaDB client library non-blocking API and libevent, using MySQL_Monitor_State_Data to track state machine (experimental, and may change a lot in the near future)
- implemented MySQL_Monitor_Connection_Pool , a new connection pool only for MySQL Monitoring
- in MySQL_Thread added variable monitor_history to define the length of logs
- in MySQL_Thread added variables monitor_username and monitor_password to define monitoring credentials
- in MySQL_Thread added variable monitor_timer_cached to define if monitoring timers can be cached
- not completed feature: MySQL_Thread added variables monitor_query_variables, monitor_query_status, monitor_query_interval and monitor_query_timeout to define how global variables and global status are collected
Admin:
- added access to monitordb, to query MySQL_Monitor tables
Drafting implementation of command line options --initial and --reload (issues #230 and #231)
Removing command line option --admin-pathdb
Cleaning ProxySQL_GlobalVariables::process_opts_pre() and ProxySQL_GlobalVariables::process_opts_post()
Standard_ProxySQL_Admin::init() now opens file GloVars.admindb instead of "proxysql.db"
Improving command line processing in main()
ProxySQL_ConfigFile::OpenFile() exits in case of parsing error
Adding docs file on command line options
Added functions:
MySQL_Protocol::generate_COM_QUERY()
MySQL_Connection::set_charset()
MySQL_Session::handler___status_CHANGING_CHARSET()
MySQL_Session::handler___client_DSS_QUERY_SENT___send_SET_NAMES_to_backend()
Added CHANGING_CHARSET in enum session_status, and handled in MySQL_Session
Still TODO:
track and handle client’s change of charset
Implemented functions:
- MySQL_HostGroups_Manager::dump_table_mysql_servers()
- MySQL_HostGroups_Manager::generate_mysql_servers_table()
- Standard_ProxySQL_Admin::save_mysql_servers_runtime_to_database() (issue #17)
Improved Standard_ProxySQL_Admin::load_mysql_servers_to_runtime() to process also status
Improved MySQL_HostGroups_Manager::commit() to modified MySrvCs
Database in MyHGM is accessible via Admin (issue #212)
Enforced debugging if specified at the command line (issue #213)