Functions add() and lookup() in MySQL_LDAP_Authentication have support for backend_username.
Added mysql_ldap_mapping table.
Created Admin::init_ldap() to be called after LDAP initialization.
Added better LDAP caching.
LOAD LDAP MAPPING TO RUNTIME cleans part of the cache (association to backend user).
All queries will have a comment "proxysql-ldap-user=%s" to track original user
The current parser for SET in MySQL_Session is not able to parse multiple
variables SET commands like:
SET sql_mode='TRADITIONAL', NAMES utf8 COLLATE unicode_ci
This patch introduces a simple regex based parser for all variation of
simple variables.
This is not a generic SET parser, though.
Fixed STMT_SEND_LONG_DATA processing which was incorrectly reading data from the STMT_EXECUTE packet, causing corruption of any subsequent parameters.
Prior to this it was only safe to use STMT_SEND_LONG_DATA for the last parameter of the statement.
Also:
* Corrected some comment indentation.
Fixes: #1283
The fix for bug #1038 was to not return a connection to the connection pool if it has an error.
Although this is correct, it also has the side effect that connections coming
from connection pool and failing during the first query because the connection
was already broken, would be considered as possibly to run a transaction.
That is incorrect.
Now at connection level it is tracked if the transaction status is known or not.
Variable reset_connection_algorithm could either be:
1 = algorithm used up too version 1.4
2 = algorithm new since ProxySQL 2.0 (now default)
When reset_connection_algorithm = 2 , MySQL_Thread itself tries to reset connections instead of relying on connections purger HGCU_thread_run()
* is STMT_PREPARE failed when there is a previous status (maybe STMT_EXECUTE) disconnect the client immediately. Maybe this is related to #1366 , but I can't reproduce it yet
* purge prepared statements is also their server reference counter is 0
* initialize connections queue for purging connections
Statistics collected aboout GTID queries
Statistics displayed on HTTP server
Count number of GTID events per server
Online upgrade of all mysql_connections tables
Fixed path for libev
Handling of prepared statements changed a lot in 1.4 , as a lot of code was rewritten.
Old code was still present, and was possible to toggle it on and off based on PROXYSQL_STMT_V14 .
Because only the new code is maintained, all references to old code are now removed,
including PROXYSQL_STMT_V14
If variable mysql-verbose_query_error is set, "Error during query" message will be extended adding:
- username
- client IP
- schemaname
- digest of the original query (not the original query itself)
If autocommit_false_is_transaction=true (false by default), a connection
with autocommit=0 is treated as a transaction.
If forward_autocommit=true (false by default), the same behavior applies.
3 server errors were retried only once, and then destroying the client connection instead of returning error:
* case 1290: // read-only
* case 1047: // WSREP has not yet prepared node for application use
* case 1053: // Server shutdown in progress
This fix should also improve the handling of graceful shutdown (error 1053)
Introduced 2 new global variables:
* mysql-stats_time_backend_query (default true)
* mysql-stats_time_query_processor (default true)
For backward compatibility, they are both enabled by default
Added variable for SQLite3 Server
Added new command:
* LOAD SQLITESERVER VARIABLES FROM MEMORY / TO RUNTIME
* LOAD SQLITESERVER VARIABLES FROM DISK / TO MEMORY
* SAVE SQLITESERVER VARIABLES FROM RUNTIME / TO MEMORY
* SAVE SQLITESERVER VARIABLES FROM MEMORY / TO DISK
Connections to SQLite3 Server uses the same MySQL users in `mysql_users`
Fixed minor issues related to ClickHouse Serve
Fixed also some Makefile errors
If a user is configured with fast_forward, no connection should be taken from the connection pool.
Yet the Hostgroup Manager should know about such connection.
Extended class ProxySQL_Checksum_Value() in ProxySQL_Cluster module to support further metrics
Implemeted `SELECT GLOBAL_CHECKSUM()` and relative tracking of global checksums
Added variable `admin-cluster_check_status_frequency` to check peer's global status at regular intervals
Automatically reset mysql->insert_id when a connection is sent to connection pool.
If is not reset to 0 , when a backend connections is assigned to a client, this one can potentially read the last inserted id from the previous client.
Extending mysql_query_rules with a new field: OK_msg
If OK_msg is not NULL, an OK packet is sent to the client with an optional
message as specified in OK_msg itself.
If OK_msg is an not NULL empty string, an OK packet is sent to the client
without any message.
If both error_msg and OK_msg are present, error_msg is returned.
This commit includes also few minor bugs, mostly related to typo that would
prevent online upgrade of mysql_query_rules from early release of 1.4.0 .
If the users connection is rejected we shouldn't be incrementing
`num_connections_used` on the account_details object. This leads to
`free_users` dropping below zero and _all_ subsequent requests getting
rejected for that user.
Refs #940
If transaction_persistent is enabled and some query would disable multiplexing,
for example "SELECT @@hostname", query routing was still disabled even after
the transaction has committed
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
These improvements avoid a race condition in which a prepared statement is removed from the cache before it is marked as used by the client.
This race condition was noticed when running benchmark creating 7000 _unique_ prepared statements per second.
Metrics introduced:
* ConnPool_get_conn_failure : connection pool cannot provide any connection
* ConnPool_get_conn_immediate : connection is provided from per-thread cache
* ConnPool_get_conn_success : the session is able to get a connection, either from per-thread cache or connection pool
- default_hostgroup was not set: this causes multiplexing to always be disabled
- fd in client_myds for mirror session was not initialized: this causes random crashes
- default_hostgroup was not set: this causes multiplexing to always be disabled
- fd in client_myds for mirror session was not initialized: this causes random crashes
MySQL_STMTs_meta() is aware to which session it belongs
entries are removed from MySQL_STMTs_meta() when unused
find_prepared_statement_by_hash() increases the client ref count
Hardcoded a maximum number of statements
The called of MySQL_STMTs_local functions doesn't must specify if it is a client or a backend.
Added also a function in MySQL_Connection to make the connection aware it is a client.
The current version of ProxySQL filters SHOW WARNINGS returning
an empty resultset.
This implementation is logically incorrect, and in future version a better
implementation will be developed
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
These are 2 memory leaks related to prepared statements
* Resultset (MySQL_ResultSet() not deallocated
* mysql_stmt_free_result() is now correctly called
So far, single thread sysbench is working.
Multi-threaded and resume from broken connections is still not working.
It means it still needs a lot of changes.
A new Query_Processor_Output (qpo) is allocated for every query.
Allocation is now done once per session, thus speeding up execution time
under memory profiler
Issue #527
Limit the amount of data received by a backend during a single poll() iteration.
Right now the default is mysql_thread___threshold_resultset_size*2, and it is arbitary.
Unrelated to #527, MySQL_Connection::async_ping() was modified to return a different value in case of timeout
jemalloc configured with purge:decay_time
Temporary disabled jemalloc to test a heap profiler (heaptrack)
MySQL_ResultSet::buffer_to_PSarrayOut() calls realloc() if the buffer preallocated is too big
MySQL_Session::writeout() sends as much data as possible to the client
This variable limits the amount of data that the client can send to the backend.
Note that ProxySQL won't really limit the amount of data that the client can send to the proxy, but the amount of data that the prox can send to the backend.
This commits also fixes issue #526Closes#526
When a server isn't responding to ping, it is flagged as shunned and all the connections need to be dropped.
All the connections must be dropped before the server is brought back online
Some legacy code is not currently used:
- removed reference to MySQL_Data_Stream::move_from_OUT_to_OUTpending()
- removed reference to MySQL_Data_Stream::PSarrayOUTpending
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules
Add new variable MySQL_Session::mirror (false by default)
Add new struct MySQL_Session::mirrorPkt (false by default)
These functions return immediately if this is a mirror session:
* MySQL_Protocol::generate_pkt_EOF()
* MySQL_Protocol::generate_pkt_ERR()
* MySQL_Protocol::generate_pkt_OK()
* MySQL_Protocol::generate_pkt_column_count()
* MySQL_Protocol::generate_pkt_field()
* MySQL_Protocol::generate_pkt_row3()
MySQL_ResultSet::MySQL_ResultSet() exits almost immediately if this is a mirror session
Most of the code of these functions is not executed if this is a mirror session:
* MySQL_ResultSet::add_eof()
* MySQL_ResultSet::get_resultset()
In MySQL_Session::writeout() :
* `client_myds->write_to_net_poll()` is called only if this is not a mirror session
In MySQL_Session::handler():
* if query is a `SELECT` (hardcoded for now) :
** create a new session and mark it as `mirror=true`
** duplicate the query and send it to the new session
* if `mirror==true` :
** pretend to read a new query from c`lient_myds`
** proceed normally (with the only exception that no data can be sent to a client)
In MySQL_Thread::process_all_sessions() :
* if `mirror==true` and `status==WAITING_CLIENT_DATA` it means the session is "completed" so it get destroyed
Limitations:
* it currently doesn't support large packets
* doesn't support different charset
* doesn't support transactions
Moved some code from MySQL_Session:handler_special_queries() into two new functions:
- MySQL_Session::handler_CommitRollback()
- MySQL_Session::handler_SetAutocommit()
These two new functions are called nowhere: de facto they are disabled
The follow states are disabled in Admin module is the backend:
- CHANGING_USER_SERVER
- CHANGING_SCHEMA
- CHANGING_CHARSET
- CHANGING_AUTOCOMMIT
- mysql-monitor_username is always added as a possible user : this allows MySQL_Monitor to monitor the backends
- admin-admin_credentials and admin-stats_credentials are always added as possible users
- mysql-monitor_username , admin-admin_credentials and stats-admin_credentials do not shows in mysql_users table (by design)
- the above users have a max_connections limit of 1000 (hardcoded)
In order to understand the performance improvement related to how ProxySQL filters autocommit/commit/rollback, the follows variables were added in GLOBAL MYSQL STATUS
- Com_autocommit
- Com_autocommit_filtered
- Com_commit
- Com_commit_filtered
- Com_rollback
- Com_rollback_filtered
- Re-enabled MySQL_Logger
- handling special case in which mysql-commands_stats and mysql-query_digests are disabled
- specifies the length of the query
- Query_Info is always initialized
Summary:
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
Query Cache enhancement on user/schema
Query Cache now supports distinct entries based on user/schema .
This is achieved passing to Query_Cache::set() and Query_Cache::get() the hash that identifies user and schema
Test Plan: None
Reviewers: rene
Differential Revision: http://phab.sysown.com/D3
removed KV_Btree_Array.cpp
removed query_SQL from MySQL_Data_Stream
added some documentation for QC_entry_t
Query_Cache::get() and Query_Cache::set() now requires also the length of the key
Query_Cache::get() and Query_Cache::set() now process hk (the hash key) using the length of the key
- MySQL_Session::handler_special_queries() processes set autocommit from clients
- to the client is always returned the autocommit value set in the session
- set autocommit is either just acknowledged to the client, or forwarded to the backend based on the status of transactions
- unless there are query rules to handle SET AUTOCOMMIT , the statement is forwarded to the first found active transaction: this assumes there is just one transactions. Although multiple transactions are possible, handling multiple transactions at the same time seems a dangerous adventure
Added MySQL_Session::RequestEnd() to wrap CurrentQuery.end() and other state changes : this to minimize bugs
MySQL_Session::RequestEnd() sets query end time instead of Query_Info::end()
MySQL_Session::RequestEnd() is called also in case of special queries handled by handler_special_queries()