The 2 new variables are:
* mysql-query_digests_max_digest_length : defines the maximum length of digest_text as reported in stats_mysql_query_digest
* mysql-query_digests_max_query_length : defines the maximum query length processed when computing query's digest and digext_text
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
This commit is able to speed up LOAD MYSQL SERVERS TO RUNTIME up to 10x for very large mysql_servers table with tens of thousands of servers.
Main improvements:
* removed 1 unnecessary call to `generate_mysql_servers_table()`
* replaced 1 call to `generate_mysql_servers_table()` with UPDATE statements when necessary
* implemented new function `servers_add()` that replaces `server_add()` to perform bulk operation
* replaced calls to `sqlite3_exec()` with proper parameters bindings
* use of multirows INSERT statements (32 rows at the time)
Miror improvements:
* added new variable mysql-hostgroup_manager_verbose , 1 by default for backward compatibility and debugging
* in not debug mode, access to HGM db has no shared cache
* in not debug mode, tables in HGM have checks disabled
Further note:
* Once issue #797 is completed, `mysql_servers` in MyHGM can be simplified further
These improvements avoid a race condition in which a prepared statement is removed from the cache before it is marked as used by the client.
This race condition was noticed when running benchmark creating 7000 _unique_ prepared statements per second.
Monitor threads are started:
* without arena cache
* with 64KB stack
malloc_conf changed from:
xmalloc:true,lg_tcache_max:16,purge:decay
to:
xmalloc:true,lg_chunk:18,lg_tcache_max:12,purge:ratio
Conflicts:
lib/MySQL_Monitor.cpp
Depending from the query, an error "Query execution was interrupted" can be returned either as:
* a standalone ERR Packet (previously handled)
* an ERR Packet at the end of a result set : this commit handle this case
Metrics introduced:
* ConnPool_get_conn_failure : connection pool cannot provide any connection
* ConnPool_get_conn_immediate : connection is provided from per-thread cache
* ConnPool_get_conn_success : the session is able to get a connection, either from per-thread cache or connection pool
MySQL_STMTs_meta() is aware to which session it belongs
entries are removed from MySQL_STMTs_meta() when unused
find_prepared_statement_by_hash() increases the client ref count
Hardcoded a maximum number of statements
The called of MySQL_STMTs_local functions doesn't must specify if it is a client or a backend.
Added also a function in MySQL_Connection to make the connection aware it is a client.
New variable admin-hash_passwords , default true
When set, passwords in mysql_users are automatically hashed in runtime
when running LOAD MYSQL USERS TO RUNTIME.
For backward compatibility, passwords in mysql_users in memory and disk aren't
automatically hashed. Although, they can be easily hashed running
SAVE MYSQL USERS TO MEMORY (and eventually SAVE MYSQL USERS TO DISK)
immediately after running LOAD MYSQL USERS TO RUNTIME
If MHM_PTHREAD_MUTEX is defined MySQL_HostGroups_Manager.h ,
MySQL_HostGroups_Manager() is compiled using pthread mutex instead od spin loop.
As a lot of computation is protected by mutex, maybe it is better not to use spin loop.
Command "LOAD SCHEDULER FROM CONFIG" is now implemented.
Config file can also be read with --initial, --reload, and when a db file is not present yet.
An example was added in src/proxysql.cfg
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
When query rules are processed, even if flagOUT is set rules are processed
in numerical order.
This new variable allows to jump back (reset rule_id to 0) N amount of times.
Using this variable is possible to implement loops.
Futhermore, a loop is possible with only one rule setting flagIN=flagOUT .
The default is 0. The maximum value is 1000000 .
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
So far, single thread sysbench is working.
Multi-threaded and resume from broken connections is still not working.
It means it still needs a lot of changes.
A new Query_Processor_Output (qpo) is allocated for every query.
Allocation is now done once per session, thus speeding up execution time
under memory profiler
This feature is not completed yet.
libmariadb/password.c is patched to use either password or sha1(password)
Class MySQL_Connection_userinfo() is expanded to also support sha1_pass
Inroduced MySQL_Authentication::set_SHA1() to store this information at runtime only
Added several auxiliary functions
This variable limits the amount of data that the client can send to the backend.
Note that ProxySQL won't really limit the amount of data that the client can send to the proxy, but the amount of data that the prox can send to the backend.
This commits also fixes issue #526Closes#526
Scheduler is being implemented as part of the Admin module.
Added table scheduler
Added table runtime_scheduler
Implemented commands:
- [LAD|SAVE] SCHEDULER [FROM|TO] [DISK|MEMORY|RUNTIME]
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
- added new function realtime_time() to get real time
- added support for multiple events, default PROXYSQL_QUERY
- MySQL_Event::write() will perfom a different action depending from log_event_type
- rewrote part of eventslog_reader_sample.cpp:
- it had few bugs
- MySQL_Event::read() will perform a different action depending from log_event_type
- time are printed in real time and not monotonic time
- all info about a query are printed in one line
These two new columns will be used to:
- enable SSL on a per server basis
- define maximum latency for a server to be consider usable: this will be a key element for multi DC support
When a server isn't responding to ping, it is flagged as shunned and all the connections need to be dropped.
All the connections must be dropped before the server is brought back online
mysql_query_rules table is changing from version to version.
In order to provide online upgrade, a new function was added:
- ProxySQL_Admin::disk_upgrade_mysql_query_rules()
Some legacy code is not currently used:
- removed reference to MySQL_Data_Stream::move_from_OUT_to_OUTpending()
- removed reference to MySQL_Data_Stream::PSarrayOUTpending
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules
Add new variable MySQL_Session::mirror (false by default)
Add new struct MySQL_Session::mirrorPkt (false by default)
These functions return immediately if this is a mirror session:
* MySQL_Protocol::generate_pkt_EOF()
* MySQL_Protocol::generate_pkt_ERR()
* MySQL_Protocol::generate_pkt_OK()
* MySQL_Protocol::generate_pkt_column_count()
* MySQL_Protocol::generate_pkt_field()
* MySQL_Protocol::generate_pkt_row3()
MySQL_ResultSet::MySQL_ResultSet() exits almost immediately if this is a mirror session
Most of the code of these functions is not executed if this is a mirror session:
* MySQL_ResultSet::add_eof()
* MySQL_ResultSet::get_resultset()
In MySQL_Session::writeout() :
* `client_myds->write_to_net_poll()` is called only if this is not a mirror session
In MySQL_Session::handler():
* if query is a `SELECT` (hardcoded for now) :
** create a new session and mark it as `mirror=true`
** duplicate the query and send it to the new session
* if `mirror==true` :
** pretend to read a new query from c`lient_myds`
** proceed normally (with the only exception that no data can be sent to a client)
In MySQL_Thread::process_all_sessions() :
* if `mirror==true` and `status==WAITING_CLIENT_DATA` it means the session is "completed" so it get destroyed
Limitations:
* it currently doesn't support large packets
* doesn't support different charset
* doesn't support transactions
All save commands now make sure that the configs are first copied from
runtime to memory before theyre read and that they are copied from
memory to runtime after they are written
Related to bug https://mariadb.atlassian.net/browse/MDEV-8338 .
Created new variable: mysql-bug_mdev_8338
If this variable is set, multiplexing is disabled if a query has NOW() or CURDATE()
Moved some code from MySQL_Session:handler_special_queries() into two new functions:
- MySQL_Session::handler_CommitRollback()
- MySQL_Session::handler_SetAutocommit()
These two new functions are called nowhere: de facto they are disabled
Features added:
* added new global variable admin-read_only: it is possible to enable read_only changing this variable and running LOAD ADMIN VARIABLES TO RUNTIME
* admin interface now replies to SHOW GLOBAL VARIABLES LIKE 'read_only'
* added new command PROXYSQL READONLY to enable read only mode
* added new command PROXYSQL READWRITE to disable read only mode : this is the only way to disable read only mode while global_variables is not readable
Note that this doesn't prevent the execution of commands LOAD and SAVE : these commands need to be used with caution
In order to understand the performance improvement related to how ProxySQL filters autocommit/commit/rollback, the follows variables were added in GLOBAL MYSQL STATUS
- Com_autocommit
- Com_autocommit_filtered
- Com_commit
- Com_commit_filtered
- Com_rollback
- Com_rollback_filtered
Summary:
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Query Cache enhancement on user/schema
Query Cache now supports distinct entries based on user/schema .
This is achieved passing to Query_Cache::set() and Query_Cache::get() the hash that identifies user and schema
Test Plan: None
Reviewers: rene
Differential Revision: http://phab.sysown.com/D3
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Added C++ headers file in cpp.h
bug fix in xchg_32 : gcc ignores it, clang doesn't
Replaced -rdynamic with --export-dynamic
Removed the override of env variables CC , CXX , CFLAGS , CPPFLAGS and LIBS
Fixed definition in MySQL_Monitor.cpp : gcc ignores it, clang doesn't
Added a compile example text file
- MySQL_Session::handler_special_queries() processes set autocommit from clients
- to the client is always returned the autocommit value set in the session
- set autocommit is either just acknowledged to the client, or forwarded to the backend based on the status of transactions
- unless there are query rules to handle SET AUTOCOMMIT , the statement is forwarded to the first found active transaction: this assumes there is just one transactions. Although multiple transactions are possible, handling multiple transactions at the same time seems a dangerous adventure
Added MySQL_Session::RequestEnd() to wrap CurrentQuery.end() and other state changes : this to minimize bugs
MySQL_Session::RequestEnd() sets query end time instead of Query_Info::end()
MySQL_Session::RequestEnd() is called also in case of special queries handled by handler_special_queries()
class MySQL_HostGroups_Manager() has now also access to admindb
added function MySQL_HostGroups_Manager::purge_mysql_servers_table() to remove entries from mysql_servers table from within MyHGM
purge_mysql_servers_table() and generate_mysql_servers_table() are called in multiple part of the code to get consistent view of mysql backends
added function MySQL_HostGroups_Manager::read_only_action(char*,int,int) responsible for triggering a traffic switch
Added 2 new functions to answer some of queries from mysqldump:
ProxySQL_Admin::generate_show_fields_from()
ProxySQL_Admin::generate_show_table_status()
ProxySQL_Admin() replies with an OK or an artificial resultset to most of the queries from mysqldump
Added status MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG as internal only (not configurable)
Added MySQL_HostGroups_Manager::replication_lag_action() to communicate replication lag to connection pool and take action
monitor_replication_lag_interval
monitor_replication_lag_timeout
Drafting MySQL_Monitor::monitor_replication_lag() as a clone of MySQL_Monitor::monitor_ping()
Status variables implemented so far:
- Client_Connections_aborted
- Client_Connections_connected
- Client_Connections_created
- Questions
- Slow_queries
Also created alias command SHOW MYSQL STATUS
- added global variables mysql-query_digests
- initialization if either mysql_thread___commands_stats or mysql_thread___query_digests are enabled
- query_parser_free() checks if QueryParserArgs was initialized
- Query_Processor::query_parser_init() checks the command type only if mysql_thread___commands_stats is true
- Query_Processor::query_parser_init() run mysql_query_digest() only if mysql_thread___query_digests is true
- Query_Processor::query_parser_update_counters() updates query digests only if digest_text is not null
- start time is always update (this is relevant for issue #327)
SET NAMES sent from the application only change the charset in the client connection.
When a backend connection is taken from the connection pool and charset mismatches, SET NAMES is sent to the backend
- Replaced some proxy_error() with daemon_log()
- glovars.proxy_restart_on_error is configurable via command line
- glovars.proxy_restart_delay is hardcoded to 1 second
- disabled listeners on telnet
- Option --admin-socket overwrite the default admin socket
signal_all_threads() now accepts an argument (default 0). If the argument is greater than 0, the threads release the thread_mutex, and sleep for the specified millisecond .
In this way processlist will immediately access all threads
If the number of free connections is greater than free_connections_pct percentage of max_connections for that specific host,
all connections idle for longer than ping_interval_server are closed instead of sending PING
- added variables mysql_thread___connect_retries_delay and mysql_thread___sessions_sort
- sessions that cannot get a connections are sorted
- commands_stats now defaults to true
Implemented LOAD MYSQL QUERY RULES FROM CONFIG via ProxySQL_Admin::Read_MySQL_Query_Rules_from_configfile()
Fixed bug in ProxySQL_Admin::save_mysql_query_rules_from_runtime()
To handle connect timeout:
- added MySQL_Data_Stream::destroy_MySQL_Connection()
- new async status: ASYNC_CONNECT_TIMEOUT
- "select @@version_comment limit 1" is answered by proxysql and not forwarded to any backend
- added MySQL_Data_Stream variables connect_retries_on_failure and max_connect_time
- added global variables mysql-connect_timeout_server_max and mysql-connect_retries_on_failure
Improvement: proxysql returns more info in COM_STATISTICS: clients connected and uptime
Simplified the ping within MySQL_Thread
SQLite3_row::add_fields() to handle empty string
prot_status was not initialized in MySQL_Protocol
Session's default schema was always mysql_thread___default_schema , now fixed with correct value
default_schema is now read from config file
Dependencies:
- added libevent
- added mariadb-connector-c for non-blocking API
- added patch for mariadb-connector-c ( see https://mariadb.atlassian.net/browse/CONC-131 )
MySQL Monitoring:
- added table mysql_server_connect_log
- added table mysql_server_ping_log
- added access to admindb
- implemented monitoring using MariaDB client library non-blocking API and libevent, using MySQL_Monitor_State_Data to track state machine (experimental, and may change a lot in the near future)
- implemented MySQL_Monitor_Connection_Pool , a new connection pool only for MySQL Monitoring
- in MySQL_Thread added variable monitor_history to define the length of logs
- in MySQL_Thread added variables monitor_username and monitor_password to define monitoring credentials
- in MySQL_Thread added variable monitor_timer_cached to define if monitoring timers can be cached
- not completed feature: MySQL_Thread added variables monitor_query_variables, monitor_query_status, monitor_query_interval and monitor_query_timeout to define how global variables and global status are collected
Admin:
- added access to monitordb, to query MySQL_Monitor tables
Drafting implementation of command line options --initial and --reload (issues #230 and #231)
Removing command line option --admin-pathdb
Cleaning ProxySQL_GlobalVariables::process_opts_pre() and ProxySQL_GlobalVariables::process_opts_post()
Standard_ProxySQL_Admin::init() now opens file GloVars.admindb instead of "proxysql.db"
Improving command line processing in main()
ProxySQL_ConfigFile::OpenFile() exits in case of parsing error
Adding docs file on command line options
Added functions:
MySQL_Protocol::generate_COM_QUERY()
MySQL_Connection::set_charset()
MySQL_Session::handler___status_CHANGING_CHARSET()
MySQL_Session::handler___client_DSS_QUERY_SENT___send_SET_NAMES_to_backend()
Added CHANGING_CHARSET in enum session_status, and handled in MySQL_Session
Still TODO:
track and handle client’s change of charset
Implemented functions:
- MySQL_HostGroups_Manager::dump_table_mysql_servers()
- MySQL_HostGroups_Manager::generate_mysql_servers_table()
- Standard_ProxySQL_Admin::save_mysql_servers_runtime_to_database() (issue #17)
Improved Standard_ProxySQL_Admin::load_mysql_servers_to_runtime() to process also status
Improved MySQL_HostGroups_Manager::commit() to modified MySrvCs
Database in MyHGM is accessible via Admin (issue #212)
Enforced debugging if specified at the command line (issue #213)
Class Standard_MySQL_Thread was restructured
mysql_thread___ thread variables are moved out to global variables
Standard_Proxy_Admin calls Standard_MySQL_Thread::refresh_variables after initializing the thread structure
For better debugging, added functions:
MySQL_Data_Stream::clean_net_failure()
MySQL_Data_Stream::set_net_failure()
MySQL_Data_Stream::setDSS_STATE_QUERY_SENT_NET()
MySQL_Session::set_unhealthy()
Added condition mybe->server_myds->PSarrayOUTpending->len==0 to determine if DSS can be STATE_QUERY_SENT_NET
Developed class StatCounters to keep time series counters
Added 2 new instances of StatCounters:
- MySQL_Session::command_counters;
- ProxySQL_Poll::loop_counters;
Counters are dispayed in "dbg thread status"
Defined new variables mysql-connect_timeout_server_error and mysql-connect_timeout_server (issue #202 and #203) .
They are defined but not implemented yet.
Fixed two crashing bug when "gdb command" was trying to access:
- its own Query_Info if this one was not initialized
- Query_Info of another thread if this one was not initialized
Introduced new variable ProxySQL_Poll:poll_timeout that Session can overwrite to apply a timeout different than mysql_thread___poll_timeout
Added variable MySQL_Connection::last_time_used , updated at creation time and when calling push_MyConn_to_pool().
for each Connection add a "last_time_used"
Added function get_multiple_idle_connections()
Implementing Standard_MySQL_Threads_Handler::connection_manager_thread() , but doesn't seems to be a good idea
====
New attempt for connection handling
MySQL_Session has a new bool variable : connections_handler
Standard_MySQL_Thread has a new variable PtrArray : mysql_sessions_connections_handler
Added new function Standard_MySQL_Thread::register_session_connection_handler
SESSIONS_FOR_CONNECTIONS_HANDLER sessions are created in Standard_MySQL_Thread::init()
Added new function Standard_MySQL_Thread::process_all_sessions_connections_handler()
New variables in Standard_MySQL_Thread:
MySQL_Connection **my_idle_conns;
bool processing_idles;
unsigned long long last_processing_idles;
Added DSS STATE_PING_SENT
MySQL_Session::handler___status_WAITING_SERVER_DATA___STATE_PING_SENT
Removed MySQL_Connection *myconn from MySQL Backend
Removed server_myds and server_myds from MySQL_Session
Fixed a bug that caused an infinite loop in case the backed was not able to create new connection
Standard_MySQL_Threads_Handler::SQL3_Threads_status() (still not completed) now locks the MySQL Threads
Some functions of Standard_MySQL_Threads_Handler and Standard_MySQL_Thread are taken out from the class definition
In order to facilitate the reuse of code in Standard_ProxySQL_Admin.cpp :
- moved remove_spaces() from Standard_ProxySQL_Admin.cpp to gen_utils.cpp
- moved Standard_ProxySQL_Admin::SQLite3_to_MySQL() to MySQL_Session::SQLite3_to_MySQL()
Moved few macros from Standard_Query_Processor.cpp to proxysql_macros.h
Added functions:
- MySQL_Session::SQL3_Session_status()
- MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_debug()
- Standard_MySQL_Thread::SQL3_Thread_status()
- MySQL_Threads_Handler::SQL3_Threads_status() (not completed)
Modified class Query_Info to copy queries : this can be a small performance penalty, but can improve functionalities
Added mysql commands:
- DBG SESSION STATUS
- DBG THREAD STATUS
- DBG THREADS STATUS
Added variable MySQL_Thread::thread_mutex , a mutex to protect the thread: this will be used to implement access to MySQL Thread via Admin
MySQL_Thread now have a new per thread variable updated via Admin Interface :
- mysql_thread___session_debug
ProxySQL_Poll now counts number of poll() in variable "loops"
If mysql_thread___session_debug is enabled and a query starts with "dbg " it is interpreted as a command for proxysql.
In Standard_ProxySQL_Admin added a variable admin_mutex to protect the update of statistics.
Fixed minor bugs.
For issue #187:
Introduced the follow functions:
- MySQL_Session::handler___status_CHANGING_USER_SERVER()
- MySQL_Session::handler___client_DSS_QUERY_SENT___send_CHANGE_USER_to_backend
- MySQL_Protocol::generate_COM_CHANGE_USER()
Modified MySQL_Session::handler___status_CHANGING_SCHEMA() to return a boolean to handle failures
Connections where INIT_DB or COM_CHANGE_USER fails are not reusable
Added session_statuses CHANGING_USER_SERVER and CHANGING_USER_CLIENT (currently not used, but needed for issue #187)
For issue #188:
Added variable MySQL_Connection_userinfo::hash : this is compared into MySQL_Session
Added function MySQL_Connection_userinfo::compute_hash() to automatically compute MySQL_Connection_userinfo::hash when a variable is changed
hash is compute with SpookyHash::Hash64()
Fixed bug #189:
removed an parameters incorrectly passed to fprintf
In MySQL_HostGroups_Manager:
- added function add() for MySrvC
- fixed few typos : poll => pool
- added locking in several functions
First cleanup of MySQL_Hostgroup_Entry
In MySQL_Protocol:
- added variable prot_status
- added variable fd
- added function process_pkt_EOF()
- variable parent is a MySrvC *
- extended functions pkt_ok() , pkt_end() and pkt_handshake_server() to update prot_status
- added function process_pkt_EOF()
- fixed warns to be uint16_t instead of uint64_t
In MySQL_Data_Stream:
- fixed bug on read_from_net()
- added function unplug_backend()
In MySQL_Session:
- attempting multiplexing if the connection has no active transactions and in case of SELECT statements
- destination_hostgroup returned in qpo is applied
MySQL_HostGroups_Manager merges the best of:
- MySQL_HostGroups_Handler
- MySQL_Connection_Pool
Still in development: will be shadowing the original Handler and ConnPool
Implemented new class Query_Info used in MySQL_Session
Added MYSQL_COM_QUERY_command MYSQL_COM_QUERY_COMMIT
Added virtual function Query_Processor::query_parser_update_counters()
curtime moved from local variable in Standard_MySQL_Thread::run() to public variable in MySQL_Thread , so accessible by other classes
Extended stats_mysql_commands_counters from 2 columns to 15 columns!! It nows have betters statistics on timing and distribution
Added command MYSQL_COM_QUERY___NONE as a marker . It also have the special meaning that Query_Info is not a query.
commands_counters is now an array of pointers to Command_Counter objects
MySQL_Data_Stream::read_from_net() updates mypolls->last_recv[] with curtime if mypolls is active (MySQL_Thread)
MySQL_Data_Stream::write_to_net() updates mypolls->last_sent[] with curtime if mypolls is active (MySQL_Thread)
For MySQL queries (_MYSQL_COM_QUERY) queries statistics are updated via CurrentQuery, an instance of Query_Info
Implemented get_stats_commands_counters() in Query_Processor
Implemented stats___mysql_commands_counters() in ProxySQL_Admin
Added table mysql_commands_counters
Added MYSQL_COM_QUERY_SHOW_TABLE_STATUS in MYSQL_COM_QUERY_command
Standard_Query_Processor has the commands description in commands_counters_desc
Introduced per thread storage variable _thr_commands_counters
query_parser_command_type() is now a wrapper to __query_parser_command_type() so to update counters based on its return value
Bug fixed:
Added transactions in Standard_ProxySQL_Admin::stats___mysql_query_rules()
Added SHOW command in libinjection
Implemented query_parser_command_type() to at least recognize queries from sysbench
Currently commands supported:
- ALTER [ONLINE | OFFLINE] [IGNORE] TABLE
- ANALYZE [NO_WRITE_TO_BINLOG | LOCAL] TABLE
- BEGIN
- COMMIT
- DELETE
- INSERT
- SELECT (not for update)
- SET
- UPDATE
Small change on libinjection : Issue 152
Added COMMIT
Per issue #158:
- Created 4 new functions in class Query_Processor.
- See doc/internal/query_parser.txt
Per issue #152:
- Patched deps/libinjection/Makefile to compile with -fPIC
- Standard_MySQL_Thread is linked against libinjection
- created Query_Processor::update_query_processor_stats()
- every mysql_thread___poll_timeout millisecond MySQL_Threads call GloQPro->update_query_processor_stats() to update global statistics
update_query_processor_stats can be extended to load more type of queries
- changed hits from uint64_t to int;
- QP_rule_t has a pointer to its parent : this is done to speed up lookup
- implemented get_current_query_rules() to execute SAVE MYSQL QUERY RULES FROM RUNTIME (issue #93)
- implemented get_stats_query_rules() to save statistics on stats_mysql_query_rules (issue #139)
- created 2 new classes to easily process statistics in Query Processor : QP_rule_text and QP_rule_text_hitsonly
- extended classes SQLite3_row and SQLite3_result
- every __admin_refresh_interval milliseconds Admin refreshes statistics
- defined table stats_mysql_query_rules
- deprecated __attach_configdb_to_admindb() , replaced with __attach_db_to_admindb (used in issue #142)
- to admindb are now attached both configdb and statsdb (issue #142)
- drafted a way to update hits on Query Processor . This method will go away when issue #146 will be implemented
- fixed crashing bug in issue #145
- added some doc about global_variables
- added c_tokenizer to tokenize C strings. Its first use is to split user/pass via c_split_2()
Issue #127
- extensive work here , now Admin support its own global variables
- commands available are:
LOAD|SAVE ADMIN VARIABLES FROM|TO DISK|MEMORY|RUNTIME
Issue #126
- username and password to access admin interface are now configurable in global_variables
Issue #129
- started deprecating add_default_user(), deprecated by #126
Drafting Monitor interface : issue #130
Bug fixes:
- fixed typo in SAVE MYSQL VARIABLES TO DISK;
Calls from Standard_ProxySQL_Admin to MySQL_Threads_Handler are wrapped by lock
LOAD MYSQL VARIABLES TO RUNTIME (or equivalent) calls commit() so that other threads will update
New commands implemented:
SAVE MYSQL SERVERS TO MEMORY
SAVE MYSQL SERVERS TO MEM
SAVE MYSQL SERVERS FROM RUNTIME
SAVE MYSQL SERVERS FROM RUN
LOAD MYSQL SERVERS FROM MEMORY
LOAD MYSQL SERVERS FROM MEM
LOAD MYSQL SERVERS TO RUNTIME
LOAD MYSQL SERVERS TO RUN
LOAD MYSQL SERVERS TO MEMORY
LOAD MYSQL SERVERS TO MEM
LOAD MYSQL SERVERS FROM DISK
For issue #18:
Implemented functions:
Standard_ProxySQL_Admin::flush_mysql_servers__from_disk_to_memory()
Standard_ProxySQL_Admin::flush_mysql_servers__from_memory_to_disk()
Standard_ProxySQL_Admin::load_mysql_servers_to_runtime()
For issue #70:
Removed any 127.0.0.1 from main()
Probably completed
Moved around some code in main (it needs a lot of clean up, it is all test code)
Modules shutdown is performed only if they are active
If GloVars.global.nostart==true , ProxySQL doesn't fully restart even after a PROXYSQL RESTART, it will wait PROXYSQL START
Various bugs/improvements:
- in class MySQL_HostGroups_Handler, use of clear() for vectors MyHostGroups and Servers
- Better implementation of PROXYSQL START (issue #14)
Added pthread_mutex_t GloVars.global.start_mutex
mutex locked by Admin module if nostart is on
mutex released by Admin when issued PROXYSQL START
main() blocks until the mutex is locked
Issue #23
Fixed a bug that was always assigning GloVars.global.gdbg=true
Issue #53
Drafted a basic Standard_ProxySQL_Admin::send_MySQL_OK(MySQL_Protocol *myprot)
Issue #56
If verbosity is greater than 10 , proxy_debug_func() prints backtrace calling:
- backtrace
- backtrace_symbols
- abi::__cxa_demangle
Added more debug lines
Added debugging type PROXY_DEBUG_MYSQL_CONNPOOL
Flag -d can accept a integer value that become the default logging verbosity using init_debug_struct_from_cmdline()
in class MySQL_Connection_Pool:
- shared converted from int to bool
- added 2 functions MySQL_Connection_lookup()
in class MyConnArray:
- match() and add() are not private
- class MySQL_Connection_Pool is a friend class
- added function MyConn_find()
in class MySQL_Connection:
- class MyConnArray is a friend class
- MCA is private
- removed custom operator new() and delete() because MySQL_Connection_Pool can be shared
- all members don't use the custom memory allocator
- added private function is_expired . See issue #30
- member expire removed, and replaced with inserted_into_pool
Renamed MyHostGroups to MyHGH (short for MySQL Host Groups Handler):
name was conflicting with MyHostGroups in class MySQL_HostGroups
Renamed class MySQL_HostGroups to MySQL_HostGroups_Handler
For issue #19:
- create MySQL_Protocol::get_myds()
- create MySQL_Protocol::generate_pkt_row()
- modified MySQL_Session::(*admin_func)(MySQL_Session *) to MySQL_Session::(*admin_func)(MySQL_Session *,ProxySQL_Admin *,PtrSize_t *) so it can access more data
- fixed an invalid strdup in SQLite3_row::add_fields()
- added SQLite3_result::rows_count
- SQLite3DB instances in Standard_ProxySQL_Admin are now public
- create Standard_ProxySQL_Admin::SQLite3_to_MySQL()
For issue #23 :
- added __dump_pkt() in mysql_data_stream.cpp
Bug fix:
Added a missing pthread_mutex_unlock() in Admin/child_mysql()
Minor clean-ups
Modified Makefile to generate my_config.h
Removed dependency from libevent headers (still needed for memcached, to be removed)
Removed linking against libevent library
Added functions:
- Standard_ProxySQL_Admin::flush_debug_levels_mem_to_db
- Standard_ProxySQL_Admin::flush_debug_levels_db_to_mem
debug_levels is populated at start up
Added functions:
- __add_active_users
- __delete_inactive_users
- __refresh_users
Fixed small bug in SQLite3_result::add_column_definition()
MySQL Authentication module is now loaded before the Admin module
For issue #3:
- fixed serious bug
For issue #7:
- most of the functions in MySQL_Authentication rewritten to accept an extra argument of type enum cred_username_type
- Standard_MySQL_Authentication stores username/password in 2 structures (creds_group_t) for backend and frontend
For issue #4:
- removed fill_table__server_status()
- removed ADMIN_SQLITE_TABLE_MYSQL_SERVER_STATUS
- fixed ADMIN_SQLITE_TABLE_MYSQL_SERVER
For issue #5:
- created functions __insert_or_ignore_maintable_select_disktable() , __delete_disktable() , __insert_or_replace_disktable_select_maintable() and __attach_configdb_to_admindb()
For issue #6 and #7:
- table ADMIN_SQLITE_TABLE_MYSQL_USERS has fields use_ssl, backend and frontend