Set dynmic hostname/port
New commands:
* LOAD CLICKHOUSE VARIABLES FROM MEMORY / TO RUNTIME
* LOAD CLICKHOUSE VARIABLES FROM DISK / TO MEMORY
* SAVE CLICKHOUSE VARIABLES FROM RUNTIME / TO MEMORY
* SAVE CLICKHOUSE VARIABLES FROM MEMORY / TO DISK
* graceful handle of missing backend (before was just crashing)
* code cleanup
* adding an embedded SQLite3 connections inside ClickHouse Server to execute internally dummy queries
* filter of several SET commands
* support for SHOW [SESSION ](VARIABLES|STATUS) LIKE
* support for SHOW [GLOBAL|ALL] VARIABLES
* support for SHOW GLOBAL STATUS
* support for SHOW COLLATION #1136
* support for SHOW CHARSET #1136
* support for SHOW ENGINES #1139
* support for SELECT current_user() #1135
* support for SELECT CONNECTION_ID() #1133
* support for SHOW FULL TABLES FROM `default`
* working semi-support for SHOW COLUMNS FROM
If a user is configured with fast_forward, no connection should be taken from the connection pool.
Yet the Hostgroup Manager should know about such connection.
In `MySQL_Cluster` class added functions to sync with remote node, like:
- `pull_mysql_query_rules_from_peer()`
- `pull_mysql_servers_from_peer()`
- `pull_mysql_users_from_peer()`
- `pull_proxysql_servers_from_peer()`
Added 8 new global variables in Admin.
4 variables determine after how many different checks the remote configuration will be synced:
- cluster_mysql_query_rules_diffs_before_sync
- cluster_mysql_servers_diffs_before_sync
- cluster_mysql_users_diffs_before_sync
- cluster_proxysql_servers_diffs_before_sync
4 variables determine if after a remote sync the changes need to be written to disk:
- cluster_mysql_query_rules_save_to_disk
- cluster_mysql_servers_save_to_disk
- cluster_mysql_users_save_to_disk
- cluster_proxysql_servers_save_to_disk
Table `proxysql_servers` is now automatically loaded from disk to memory and into runtime at startup.
Added new Admin's command `LOAD PROXYSQL SERVERS FROM CONFIG` to load `proxysql_servers` from config file to memory (not runtime).
Internal structures with credentials in MySQL_Authentication moved from `unsorted_map` to `map` : this to ensure the right order when generating the checksum.
Config file supports both `address` and `hostname` for `mysql_servers` and `proxysql_servers` , #1091
`ProxySQL_Admin::load_proxysql_servers_to_runtime()` now has a lock or no lock option, to avoid deadlock
For now, Cluster module is quite verbose.
Extended class ProxySQL_Checksum_Value() in ProxySQL_Cluster module to support further metrics
Implemeted `SELECT GLOBAL_CHECKSUM()` and relative tracking of global checksums
Added variable `admin-cluster_check_status_frequency` to check peer's global status at regular intervals
This commit introduces:
2 new tables:
* `runtime_checksums_values` : stores checksums of configurations in runtime. For now for `mysql_query_rules`, `mysql_servers` and `mysql_users`
* `stats_proxysql_servers_checksums` : when clustering is enabled, it collects all metrics from `runtime_checksums_values` from all its peers
3 new global variables that defines it checksum needs to be generated during `LOAD ... TO RUNTIME`
* `admin-checksum_mysql_query_rules`
* `admin-checksum_mysql_servers`
* `admin-checksum_mysql_users`
ProxySQL Cluster connections now have timeouts:
* 1 second timeout for CONNECT and WRITE
* 60 seconds timeout for READ (useful for long poll)
Status of the server was not changed if:
* was OFFLINE_HARD
* was incorrectly present in myhgm.mysql_servers
Added also a mutex to serialize the read_only actions
Extending mysql_query_rules with a new field: OK_msg
If OK_msg is not NULL, an OK packet is sent to the client with an optional
message as specified in OK_msg itself.
If OK_msg is an not NULL empty string, an OK packet is sent to the client
without any message.
If both error_msg and OK_msg are present, error_msg is returned.
This commit includes also few minor bugs, mostly related to typo that would
prevent online upgrade of mysql_query_rules from early release of 1.4.0 .
If total mysql_servers.weight per hostgroup was higher than 64K, traffic was unbalanced
Conflicts:
include/gen_utils.h
lib/MySQL_HostGroups_Manager.cpp
Enforced max_writers
Enforced writer_is_also_reader
Increased the number of entries in mysql_server_group_replication_log to 100 per node
Lowered the minimum value for mysql-monitor_groupreplication_healthcheck_interval to 50ms
Lowered the minimum value for mysql-monitor_groupreplication_healthcheck_timeout to 50ms
New command `PROXYSQL FLUSH CONFIGDB` allows to close and reopen configdb (`proxysql.db` on disk).
Note:
It is important that the (new) configdb is a valid one.
A misconfigured configdb will cause undefined behaviors.
The 2 new variables are:
* mysql-query_digests_max_digest_length : defines the maximum length of digest_text as reported in stats_mysql_query_digest
* mysql-query_digests_max_query_length : defines the maximum query length processed when computing query's digest and digext_text
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
This commit is able to speed up LOAD MYSQL SERVERS TO RUNTIME up to 10x for very large mysql_servers table with tens of thousands of servers.
Main improvements:
* removed 1 unnecessary call to `generate_mysql_servers_table()`
* replaced 1 call to `generate_mysql_servers_table()` with UPDATE statements when necessary
* implemented new function `servers_add()` that replaces `server_add()` to perform bulk operation
* replaced calls to `sqlite3_exec()` with proper parameters bindings
* use of multirows INSERT statements (32 rows at the time)
Miror improvements:
* added new variable mysql-hostgroup_manager_verbose , 1 by default for backward compatibility and debugging
* in not debug mode, access to HGM db has no shared cache
* in not debug mode, tables in HGM have checks disabled
Further note:
* Once issue #797 is completed, `mysql_servers` in MyHGM can be simplified further
These improvements avoid a race condition in which a prepared statement is removed from the cache before it is marked as used by the client.
This race condition was noticed when running benchmark creating 7000 _unique_ prepared statements per second.
Monitor threads are started:
* without arena cache
* with 64KB stack
malloc_conf changed from:
xmalloc:true,lg_tcache_max:16,purge:decay
to:
xmalloc:true,lg_chunk:18,lg_tcache_max:12,purge:ratio
Conflicts:
lib/MySQL_Monitor.cpp
Depending from the query, an error "Query execution was interrupted" can be returned either as:
* a standalone ERR Packet (previously handled)
* an ERR Packet at the end of a result set : this commit handle this case
Metrics introduced:
* ConnPool_get_conn_failure : connection pool cannot provide any connection
* ConnPool_get_conn_immediate : connection is provided from per-thread cache
* ConnPool_get_conn_success : the session is able to get a connection, either from per-thread cache or connection pool
MySQL_STMTs_meta() is aware to which session it belongs
entries are removed from MySQL_STMTs_meta() when unused
find_prepared_statement_by_hash() increases the client ref count
Hardcoded a maximum number of statements
The called of MySQL_STMTs_local functions doesn't must specify if it is a client or a backend.
Added also a function in MySQL_Connection to make the connection aware it is a client.
New variable admin-hash_passwords , default true
When set, passwords in mysql_users are automatically hashed in runtime
when running LOAD MYSQL USERS TO RUNTIME.
For backward compatibility, passwords in mysql_users in memory and disk aren't
automatically hashed. Although, they can be easily hashed running
SAVE MYSQL USERS TO MEMORY (and eventually SAVE MYSQL USERS TO DISK)
immediately after running LOAD MYSQL USERS TO RUNTIME
If MHM_PTHREAD_MUTEX is defined MySQL_HostGroups_Manager.h ,
MySQL_HostGroups_Manager() is compiled using pthread mutex instead od spin loop.
As a lot of computation is protected by mutex, maybe it is better not to use spin loop.
Command "LOAD SCHEDULER FROM CONFIG" is now implemented.
Config file can also be read with --initial, --reload, and when a db file is not present yet.
An example was added in src/proxysql.cfg
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
When query rules are processed, even if flagOUT is set rules are processed
in numerical order.
This new variable allows to jump back (reset rule_id to 0) N amount of times.
Using this variable is possible to implement loops.
Futhermore, a loop is possible with only one rule setting flagIN=flagOUT .
The default is 0. The maximum value is 1000000 .
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
* converted MySQL_Authentication to use unsorted_map instead of btree_map
* Query_Cache now uses pointers to KV_BtreeArray
* removed some unnecesary headers
So far, single thread sysbench is working.
Multi-threaded and resume from broken connections is still not working.
It means it still needs a lot of changes.
A new Query_Processor_Output (qpo) is allocated for every query.
Allocation is now done once per session, thus speeding up execution time
under memory profiler
This feature is not completed yet.
libmariadb/password.c is patched to use either password or sha1(password)
Class MySQL_Connection_userinfo() is expanded to also support sha1_pass
Inroduced MySQL_Authentication::set_SHA1() to store this information at runtime only
Added several auxiliary functions
This variable limits the amount of data that the client can send to the backend.
Note that ProxySQL won't really limit the amount of data that the client can send to the proxy, but the amount of data that the prox can send to the backend.
This commits also fixes issue #526Closes#526
Scheduler is being implemented as part of the Admin module.
Added table scheduler
Added table runtime_scheduler
Implemented commands:
- [LAD|SAVE] SCHEDULER [FROM|TO] [DISK|MEMORY|RUNTIME]
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
- added new function realtime_time() to get real time
- added support for multiple events, default PROXYSQL_QUERY
- MySQL_Event::write() will perfom a different action depending from log_event_type
- rewrote part of eventslog_reader_sample.cpp:
- it had few bugs
- MySQL_Event::read() will perform a different action depending from log_event_type
- time are printed in real time and not monotonic time
- all info about a query are printed in one line
These two new columns will be used to:
- enable SSL on a per server basis
- define maximum latency for a server to be consider usable: this will be a key element for multi DC support
When a server isn't responding to ping, it is flagged as shunned and all the connections need to be dropped.
All the connections must be dropped before the server is brought back online
mysql_query_rules table is changing from version to version.
In order to provide online upgrade, a new function was added:
- ProxySQL_Admin::disk_upgrade_mysql_query_rules()
Some legacy code is not currently used:
- removed reference to MySQL_Data_Stream::move_from_OUT_to_OUTpending()
- removed reference to MySQL_Data_Stream::PSarrayOUTpending
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules
Add new variable MySQL_Session::mirror (false by default)
Add new struct MySQL_Session::mirrorPkt (false by default)
These functions return immediately if this is a mirror session:
* MySQL_Protocol::generate_pkt_EOF()
* MySQL_Protocol::generate_pkt_ERR()
* MySQL_Protocol::generate_pkt_OK()
* MySQL_Protocol::generate_pkt_column_count()
* MySQL_Protocol::generate_pkt_field()
* MySQL_Protocol::generate_pkt_row3()
MySQL_ResultSet::MySQL_ResultSet() exits almost immediately if this is a mirror session
Most of the code of these functions is not executed if this is a mirror session:
* MySQL_ResultSet::add_eof()
* MySQL_ResultSet::get_resultset()
In MySQL_Session::writeout() :
* `client_myds->write_to_net_poll()` is called only if this is not a mirror session
In MySQL_Session::handler():
* if query is a `SELECT` (hardcoded for now) :
** create a new session and mark it as `mirror=true`
** duplicate the query and send it to the new session
* if `mirror==true` :
** pretend to read a new query from c`lient_myds`
** proceed normally (with the only exception that no data can be sent to a client)
In MySQL_Thread::process_all_sessions() :
* if `mirror==true` and `status==WAITING_CLIENT_DATA` it means the session is "completed" so it get destroyed
Limitations:
* it currently doesn't support large packets
* doesn't support different charset
* doesn't support transactions