- added global variables mysql-query_digests
- initialization if either mysql_thread___commands_stats or mysql_thread___query_digests are enabled
- query_parser_free() checks if QueryParserArgs was initialized
- Query_Processor::query_parser_init() checks the command type only if mysql_thread___commands_stats is true
- Query_Processor::query_parser_init() run mysql_query_digest() only if mysql_thread___query_digests is true
- Query_Processor::query_parser_update_counters() updates query digests only if digest_text is not null
- start time is always update (this is relevant for issue #327)
- typos in stats_mysql_query_digest_reset
- stats___mysql_query_digests() is executed on demand
- dump_mysql_collations() is moved after table creation
SET NAMES sent from the application only change the charset in the client connection.
When a backend connection is taken from the connection pool and charset mismatches, SET NAMES is sent to the backend
- Replaced some proxy_error() with daemon_log()
- glovars.proxy_restart_on_error is configurable via command line
- glovars.proxy_restart_delay is hardcoded to 1 second
- disabled listeners on telnet
- Option --admin-socket overwrite the default admin socket
- telnet_admin_ifaces disabled as currently not implemented
- telnet_stats_ifaces disabled as currently not implemented
- ProxySQL_Admin::get_variable() uses a safer version of strdup()
- default config file is, in order: proxysql.cnf , proxysql.cfg , /etc/proxysql.cnf , /etc/proxysql.cfg
- default datadir moved from /var/run/proxysql to current working directory
Minor bug fixes
signal_all_threads() now accepts an argument (default 0). If the argument is greater than 0, the threads release the thread_mutex, and sleep for the specified millisecond .
In this way processlist will immediately access all threads
If the number of free connections is greater than free_connections_pct percentage of max_connections for that specific host,
all connections idle for longer than ping_interval_server are closed instead of sending PING
- added variables mysql_thread___connect_retries_delay and mysql_thread___sessions_sort
- sessions that cannot get a connections are sorted
- commands_stats now defaults to true
Implemented LOAD MYSQL QUERY RULES FROM CONFIG via ProxySQL_Admin::Read_MySQL_Query_Rules_from_configfile()
Fixed bug in ProxySQL_Admin::save_mysql_query_rules_from_runtime()
To handle connect timeout:
- added MySQL_Data_Stream::destroy_MySQL_Connection()
- new async status: ASYNC_CONNECT_TIMEOUT
- "select @@version_comment limit 1" is answered by proxysql and not forwarded to any backend
- added MySQL_Data_Stream variables connect_retries_on_failure and max_connect_time
- added global variables mysql-connect_timeout_server_max and mysql-connect_retries_on_failure
Improvement: proxysql returns more info in COM_STATISTICS: clients connected and uptime
Simplified the ping within MySQL_Thread
SQLite3_row::add_fields() to handle empty string
prot_status was not initialized in MySQL_Protocol
Session's default schema was always mysql_thread___default_schema , now fixed with correct value
default_schema is now read from config file
Dependencies:
- added libevent
- added mariadb-connector-c for non-blocking API
- added patch for mariadb-connector-c ( see https://mariadb.atlassian.net/browse/CONC-131 )
MySQL Monitoring:
- added table mysql_server_connect_log
- added table mysql_server_ping_log
- added access to admindb
- implemented monitoring using MariaDB client library non-blocking API and libevent, using MySQL_Monitor_State_Data to track state machine (experimental, and may change a lot in the near future)
- implemented MySQL_Monitor_Connection_Pool , a new connection pool only for MySQL Monitoring
- in MySQL_Thread added variable monitor_history to define the length of logs
- in MySQL_Thread added variables monitor_username and monitor_password to define monitoring credentials
- in MySQL_Thread added variable monitor_timer_cached to define if monitoring timers can be cached
- not completed feature: MySQL_Thread added variables monitor_query_variables, monitor_query_status, monitor_query_interval and monitor_query_timeout to define how global variables and global status are collected
Admin:
- added access to monitordb, to query MySQL_Monitor tables
Bug fixes:
proxysql stop/start was causing a crash when trying to read from a config file already closed
fixed a typo in server_capabilities
mysql_threads can now be changed if the MySQL module is not running
Drafting implementation of command line options --initial and --reload (issues #230 and #231)
Removing command line option --admin-pathdb
Cleaning ProxySQL_GlobalVariables::process_opts_pre() and ProxySQL_GlobalVariables::process_opts_post()
Standard_ProxySQL_Admin::init() now opens file GloVars.admindb instead of "proxysql.db"
Improving command line processing in main()
ProxySQL_ConfigFile::OpenFile() exits in case of parsing error
Adding docs file on command line options
Added functions:
MySQL_Protocol::generate_COM_QUERY()
MySQL_Connection::set_charset()
MySQL_Session::handler___status_CHANGING_CHARSET()
MySQL_Session::handler___client_DSS_QUERY_SENT___send_SET_NAMES_to_backend()
Added CHANGING_CHARSET in enum session_status, and handled in MySQL_Session
Still TODO:
track and handle client’s change of charset
Issues #215 , #216 and #217
215 : Connection pool should not return connections if server status is not ONLINE
216 : Connection pool should return connections based on weight
217 : Crash if no backend
Implemented functions:
- MySQL_HostGroups_Manager::dump_table_mysql_servers()
- MySQL_HostGroups_Manager::generate_mysql_servers_table()
- Standard_ProxySQL_Admin::save_mysql_servers_runtime_to_database() (issue #17)
Improved Standard_ProxySQL_Admin::load_mysql_servers_to_runtime() to process also status
Improved MySQL_HostGroups_Manager::commit() to modified MySrvCs
Database in MyHGM is accessible via Admin (issue #212)
Enforced debugging if specified at the command line (issue #213)
Class Standard_MySQL_Thread was restructured
mysql_thread___ thread variables are moved out to global variables
Standard_Proxy_Admin calls Standard_MySQL_Thread::refresh_variables after initializing the thread structure
For better debugging, added functions:
MySQL_Data_Stream::clean_net_failure()
MySQL_Data_Stream::set_net_failure()
MySQL_Data_Stream::setDSS_STATE_QUERY_SENT_NET()
MySQL_Session::set_unhealthy()
Added condition mybe->server_myds->PSarrayOUTpending->len==0 to determine if DSS can be STATE_QUERY_SENT_NET
Developed class StatCounters to keep time series counters
Added 2 new instances of StatCounters:
- MySQL_Session::command_counters;
- ProxySQL_Poll::loop_counters;
Counters are dispayed in "dbg thread status"
Defined new variables mysql-connect_timeout_server_error and mysql-connect_timeout_server (issue #202 and #203) .
They are defined but not implemented yet.
Fixed two crashing bug when "gdb command" was trying to access:
- its own Query_Info if this one was not initialized
- Query_Info of another thread if this one was not initialized
Introduced new variable ProxySQL_Poll:poll_timeout that Session can overwrite to apply a timeout different than mysql_thread___poll_timeout
Added variable MySQL_Connection::last_time_used , updated at creation time and when calling push_MyConn_to_pool().
for each Connection add a "last_time_used"
Added function get_multiple_idle_connections()
Implementing Standard_MySQL_Threads_Handler::connection_manager_thread() , but doesn't seems to be a good idea
====
New attempt for connection handling
MySQL_Session has a new bool variable : connections_handler
Standard_MySQL_Thread has a new variable PtrArray : mysql_sessions_connections_handler
Added new function Standard_MySQL_Thread::register_session_connection_handler
SESSIONS_FOR_CONNECTIONS_HANDLER sessions are created in Standard_MySQL_Thread::init()
Added new function Standard_MySQL_Thread::process_all_sessions_connections_handler()
New variables in Standard_MySQL_Thread:
MySQL_Connection **my_idle_conns;
bool processing_idles;
unsigned long long last_processing_idles;
Added DSS STATE_PING_SENT
MySQL_Session::handler___status_WAITING_SERVER_DATA___STATE_PING_SENT
Removed MySQL_Connection *myconn from MySQL Backend
Removed server_myds and server_myds from MySQL_Session
Fixed a bug that caused an infinite loop in case the backed was not able to create new connection
Standard_MySQL_Threads_Handler::SQL3_Threads_status() (still not completed) now locks the MySQL Threads
Some functions of Standard_MySQL_Threads_Handler and Standard_MySQL_Thread are taken out from the class definition
In order to facilitate the reuse of code in Standard_ProxySQL_Admin.cpp :
- moved remove_spaces() from Standard_ProxySQL_Admin.cpp to gen_utils.cpp
- moved Standard_ProxySQL_Admin::SQLite3_to_MySQL() to MySQL_Session::SQLite3_to_MySQL()
Moved few macros from Standard_Query_Processor.cpp to proxysql_macros.h
Added functions:
- MySQL_Session::SQL3_Session_status()
- MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_debug()
- Standard_MySQL_Thread::SQL3_Thread_status()
- MySQL_Threads_Handler::SQL3_Threads_status() (not completed)
Modified class Query_Info to copy queries : this can be a small performance penalty, but can improve functionalities
Added mysql commands:
- DBG SESSION STATUS
- DBG THREAD STATUS
- DBG THREADS STATUS
Added variable MySQL_Thread::thread_mutex , a mutex to protect the thread: this will be used to implement access to MySQL Thread via Admin
MySQL_Thread now have a new per thread variable updated via Admin Interface :
- mysql_thread___session_debug
ProxySQL_Poll now counts number of poll() in variable "loops"
If mysql_thread___session_debug is enabled and a query starts with "dbg " it is interpreted as a command for proxysql.
In Standard_ProxySQL_Admin added a variable admin_mutex to protect the update of statistics.
Fixed minor bugs.
For issue #187:
Introduced the follow functions:
- MySQL_Session::handler___status_CHANGING_USER_SERVER()
- MySQL_Session::handler___client_DSS_QUERY_SENT___send_CHANGE_USER_to_backend
- MySQL_Protocol::generate_COM_CHANGE_USER()
Modified MySQL_Session::handler___status_CHANGING_SCHEMA() to return a boolean to handle failures
Connections where INIT_DB or COM_CHANGE_USER fails are not reusable
Added session_statuses CHANGING_USER_SERVER and CHANGING_USER_CLIENT (currently not used, but needed for issue #187)
For issue #188:
Added variable MySQL_Connection_userinfo::hash : this is compared into MySQL_Session
Added function MySQL_Connection_userinfo::compute_hash() to automatically compute MySQL_Connection_userinfo::hash when a variable is changed
hash is compute with SpookyHash::Hash64()
Fixed bug #189:
removed an parameters incorrectly passed to fprintf
Fixed crashing bug in Admin module when ProxySQL is started with --no-start
When a client connection is terminated, the backend connection is returned to connection pool on if not in a transaction
In MySQL_HostGroups_Manager:
- added function add() for MySrvC
- fixed few typos : poll => pool
- added locking in several functions
First cleanup of MySQL_Hostgroup_Entry
In MySQL_Protocol:
- added variable prot_status
- added variable fd
- added function process_pkt_EOF()
- variable parent is a MySrvC *
- extended functions pkt_ok() , pkt_end() and pkt_handshake_server() to update prot_status
- added function process_pkt_EOF()
- fixed warns to be uint16_t instead of uint64_t
In MySQL_Data_Stream:
- fixed bug on read_from_net()
- added function unplug_backend()
In MySQL_Session:
- attempting multiplexing if the connection has no active transactions and in case of SELECT statements
- destination_hostgroup returned in qpo is applied
MySQL_HostGroups_Manager merges the best of:
- MySQL_HostGroups_Handler
- MySQL_Connection_Pool
Still in development: will be shadowing the original Handler and ConnPool
Implemented new class Query_Info used in MySQL_Session
Added MYSQL_COM_QUERY_command MYSQL_COM_QUERY_COMMIT
Added virtual function Query_Processor::query_parser_update_counters()
curtime moved from local variable in Standard_MySQL_Thread::run() to public variable in MySQL_Thread , so accessible by other classes
Extended stats_mysql_commands_counters from 2 columns to 15 columns!! It nows have betters statistics on timing and distribution
Added command MYSQL_COM_QUERY___NONE as a marker . It also have the special meaning that Query_Info is not a query.
commands_counters is now an array of pointers to Command_Counter objects
MySQL_Data_Stream::read_from_net() updates mypolls->last_recv[] with curtime if mypolls is active (MySQL_Thread)
MySQL_Data_Stream::write_to_net() updates mypolls->last_sent[] with curtime if mypolls is active (MySQL_Thread)
For MySQL queries (_MYSQL_COM_QUERY) queries statistics are updated via CurrentQuery, an instance of Query_Info
Implemented get_stats_commands_counters() in Query_Processor
Implemented stats___mysql_commands_counters() in ProxySQL_Admin
Added table mysql_commands_counters
Added MYSQL_COM_QUERY_SHOW_TABLE_STATUS in MYSQL_COM_QUERY_command
Standard_Query_Processor has the commands description in commands_counters_desc
Introduced per thread storage variable _thr_commands_counters
query_parser_command_type() is now a wrapper to __query_parser_command_type() so to update counters based on its return value
Bug fixed:
Added transactions in Standard_ProxySQL_Admin::stats___mysql_query_rules()
Added SHOW command in libinjection
Implemented query_parser_command_type() to at least recognize queries from sysbench
Currently commands supported:
- ALTER [ONLINE | OFFLINE] [IGNORE] TABLE
- ANALYZE [NO_WRITE_TO_BINLOG | LOCAL] TABLE
- BEGIN
- COMMIT
- DELETE
- INSERT
- SELECT (not for update)
- SET
- UPDATE
Small change on libinjection : Issue 152
Added COMMIT
Per issue #158:
- Created 4 new functions in class Query_Processor.
- See doc/internal/query_parser.txt
Per issue #152:
- Patched deps/libinjection/Makefile to compile with -fPIC
- Standard_MySQL_Thread is linked against libinjection
- created Query_Processor::update_query_processor_stats()
- every mysql_thread___poll_timeout millisecond MySQL_Threads call GloQPro->update_query_processor_stats() to update global statistics
update_query_processor_stats can be extended to load more type of queries
- changed hits from uint64_t to int;
- QP_rule_t has a pointer to its parent : this is done to speed up lookup
- implemented get_current_query_rules() to execute SAVE MYSQL QUERY RULES FROM RUNTIME (issue #93)
- implemented get_stats_query_rules() to save statistics on stats_mysql_query_rules (issue #139)
- created 2 new classes to easily process statistics in Query Processor : QP_rule_text and QP_rule_text_hitsonly
- extended classes SQLite3_row and SQLite3_result
- every __admin_refresh_interval milliseconds Admin refreshes statistics
- defined table stats_mysql_query_rules
- deprecated __attach_configdb_to_admindb() , replaced with __attach_db_to_admindb (used in issue #142)
- to admindb are now attached both configdb and statsdb (issue #142)
- drafted a way to update hits on Query Processor . This method will go away when issue #146 will be implemented
- fixed crashing bug in issue #145
Now make in root directory will take care of compiling everything
DEBUG and OPTZ variables are set in root Makefile and removed from subdir
Fixed issue #137
Admin listeners are now configurable through global variables.
The 3 global variables are:
admin-mysql_ifaces
admin-telnet_admin_ifaces
admin-telnet_monitor_ifaces
It also includes some preparation for issues #132 and #133
Ensure that all the commands executed in Monitor interface are read only
Create a small lists of not allowed commands:
- PRAGMA
- ATTACH
In case such commands are executed, they are logged in error log as warnings
- added some doc about global_variables
- added c_tokenizer to tokenize C strings. Its first use is to split user/pass via c_split_2()
Issue #127
- extensive work here , now Admin support its own global variables
- commands available are:
LOAD|SAVE ADMIN VARIABLES FROM|TO DISK|MEMORY|RUNTIME
Issue #126
- username and password to access admin interface are now configurable in global_variables
Issue #129
- started deprecating add_default_user(), deprecated by #126
Drafting Monitor interface : issue #130
Bug fixes:
- fixed typo in SAVE MYSQL VARIABLES TO DISK;
Issue #123:
- in case of backend authentication failure the client is disconnected and MySQL_Connection on backend marked not reausable
Bug fixes:
- fixed a bug in MySQL_Protocol::process_pkt_OK()
- increased DEBUG_MSG_MAXSIZE : long debugging messages were causing crash
Added few more debugging entries
Issue #114 : marked with perror() + assert()
Marked with assert another not handled scenario
Calls from Standard_ProxySQL_Admin to MySQL_Threads_Handler are wrapped by lock
LOAD MYSQL VARIABLES TO RUNTIME (or equivalent) calls commit() so that other threads will update
Fixed few minor bugs.
Extended Standard_MySQL_Threads_Handler::get_variable for more variables
Better handling of threads and stacksize in Standard_MySQL_Threads_Handler::init
Implemented in Admin:
- load_mysql_variables_to_runtime
- save_mysql_variables_from_runtime
- flush_mysql_variables___database_to_runtime
- all LOAD MYSQL VARIABLES variants
- all SAVE MYSQL VARIABLES variants
New commands implemented:
SAVE MYSQL SERVERS TO MEMORY
SAVE MYSQL SERVERS TO MEM
SAVE MYSQL SERVERS FROM RUNTIME
SAVE MYSQL SERVERS FROM RUN
LOAD MYSQL SERVERS FROM MEMORY
LOAD MYSQL SERVERS FROM MEM
LOAD MYSQL SERVERS TO RUNTIME
LOAD MYSQL SERVERS TO RUN
LOAD MYSQL SERVERS TO MEMORY
LOAD MYSQL SERVERS TO MEM
LOAD MYSQL SERVERS FROM DISK
For issue #18:
Implemented functions:
Standard_ProxySQL_Admin::flush_mysql_servers__from_disk_to_memory()
Standard_ProxySQL_Admin::flush_mysql_servers__from_memory_to_disk()
Standard_ProxySQL_Admin::load_mysql_servers_to_runtime()
For issue #70:
Removed any 127.0.0.1 from main()
Implemented the follow functions:
LOAD MYSQL USERS TO MEMORY
LOAD MYSQL USERS TO MEM
LOAD MYSQL USERS FROM DISK
SAVE MYSQL USERS FROM MEMORY
SAVE MYSQL USERS FROM MEM
SAVE MYSQL USERS TO DISK
LOAD MYSQL USERS FROM MEMORY
LOAD MYSQL USERS FROM MEM
LOAD MYSQL USERS TO RUNTIME
LOAD MYSQL USERS TO RUN
Prepared the skeleton for follow functions, but the callback function is not implemented yet (issue #13):
SAVE MYSQL USERS TO MEMORY
SAVE MYSQL USERS TO MEM
SAVE MYSQL USERS FROM RUNTIME
SAVE MYSQL USERS FROM RUN
Implemented the follow commands with regards to debugging, grouped by alias:
LOAD DEBUG TO MEMORY
LOAD DEBUG TO MEM
LOAD DEBUG FROM DISK
SAVE DEBUG FROM MEMORY
SAVE DEBUG FROM MEM
SAVE DEBUG TO DISK
LOAD DEBUG FROM MEMORY
LOAD DEBUG FROM MEM
LOAD DEBUG TO RUNTIME
LOAD DEBUG TO RUN
SAVE DEBUG TO MEMORY
SAVE DEBUG TO MEM
SAVE DEBUG FROM RUNTIME
SAVE DEBUG FROM RUN
Renamed flush_debug_levels_mem_to_db to flush_debug_levels_runtime_to_database
Renamed flush_debug_levels_db_to_mem to flush_debug_levels_database_to_runtime
Probably completed
Moved around some code in main (it needs a lot of clean up, it is all test code)
Modules shutdown is performed only if they are active
If GloVars.global.nostart==true , ProxySQL doesn't fully restart even after a PROXYSQL RESTART, it will wait PROXYSQL START
Various bugs/improvements:
- in class MySQL_HostGroups_Handler, use of clear() for vectors MyHostGroups and Servers
- Better implementation of PROXYSQL START (issue #14)
Added pthread_mutex_t GloVars.global.start_mutex
mutex locked by Admin module if nostart is on
mutex released by Admin when issued PROXYSQL START
main() blocks until the mutex is locked
Issue #23
Fixed a bug that was always assigning GloVars.global.gdbg=true
Issue #53
Drafted a basic Standard_ProxySQL_Admin::send_MySQL_OK(MySQL_Protocol *myprot)
Issue #56
If verbosity is greater than 10 , proxy_debug_func() prints backtrace calling:
- backtrace
- backtrace_symbols
- abi::__cxa_demangle
Added more debug lines
Added debugging type PROXY_DEBUG_MYSQL_CONNPOOL
Flag -d can accept a integer value that become the default logging verbosity using init_debug_struct_from_cmdline()
in class MySQL_Connection_Pool:
- shared converted from int to bool
- added 2 functions MySQL_Connection_lookup()
in class MyConnArray:
- match() and add() are not private
- class MySQL_Connection_Pool is a friend class
- added function MyConn_find()
in class MySQL_Connection:
- class MyConnArray is a friend class
- MCA is private
- removed custom operator new() and delete() because MySQL_Connection_Pool can be shared
- all members don't use the custom memory allocator
- added private function is_expired . See issue #30
- member expire removed, and replaced with inserted_into_pool
Renamed MyHostGroups to MyHGH (short for MySQL Host Groups Handler):
name was conflicting with MyHostGroups in class MySQL_HostGroups
Renamed class MySQL_HostGroups to MySQL_HostGroups_Handler
For issue #19:
- create MySQL_Protocol::get_myds()
- create MySQL_Protocol::generate_pkt_row()
- modified MySQL_Session::(*admin_func)(MySQL_Session *) to MySQL_Session::(*admin_func)(MySQL_Session *,ProxySQL_Admin *,PtrSize_t *) so it can access more data
- fixed an invalid strdup in SQLite3_row::add_fields()
- added SQLite3_result::rows_count
- SQLite3DB instances in Standard_ProxySQL_Admin are now public
- create Standard_ProxySQL_Admin::SQLite3_to_MySQL()
For issue #23 :
- added __dump_pkt() in mysql_data_stream.cpp
Bug fix:
Added a missing pthread_mutex_unlock() in Admin/child_mysql()
Minor clean-ups
Added functions:
- Standard_ProxySQL_Admin::flush_debug_levels_mem_to_db
- Standard_ProxySQL_Admin::flush_debug_levels_db_to_mem
debug_levels is populated at start up
Added functions:
- __add_active_users
- __delete_inactive_users
- __refresh_users
Fixed small bug in SQLite3_result::add_column_definition()
MySQL Authentication module is now loaded before the Admin module
For issue #3:
- fixed serious bug
For issue #7:
- most of the functions in MySQL_Authentication rewritten to accept an extra argument of type enum cred_username_type
- Standard_MySQL_Authentication stores username/password in 2 structures (creds_group_t) for backend and frontend
For issue #4:
- removed fill_table__server_status()
- removed ADMIN_SQLITE_TABLE_MYSQL_SERVER_STATUS
- fixed ADMIN_SQLITE_TABLE_MYSQL_SERVER
For issue #5:
- created functions __insert_or_ignore_maintable_select_disktable() , __delete_disktable() , __insert_or_replace_disktable_select_maintable() and __attach_configdb_to_admindb()
For issue #6 and #7:
- table ADMIN_SQLITE_TABLE_MYSQL_USERS has fields use_ssl, backend and frontend