Merge branch 'v1.3.0' into v1.3.1-20161127

Conflicts:
	lib/mysql_connection.cpp
pull/827/head
René Cannaò 10 years ago
commit afea6885b8

@ -10,7 +10,7 @@ DEBUG=${ALL_DEBUG}
#export DEBUG
#export OPTZ
#export EXTRALINK
CURVER=1.3.0f
CURVER=1.3.0g
MAKEOPT="-j 8"
DISTRO := $(shell gawk -F= '/^NAME/{print $$2}' /etc/os-release)
ifeq ($(wildcard /usr/lib/systemd/systemd), /usr/lib/systemd/systemd)

@ -0,0 +1,97 @@
# ProxySQL v1.3.0 , v1.3.0f v1.3.0g
## ProxySQL v1.3.0
Release date: 2016-10-19
### Performance improvement
* support for millions of connections. It is also able to handle workloads with hundreds thousands client connections, but only few connections are active. For example, 100k total connections, but only 100 active connections.
* use of pthread_mutex for connection pool
* several performance improvements
### Usability improvement
* Packaging: version number now includes commit hash
* Monitor: limit the number of Monitor thread to 16
### New features
* EXPERIMENTAL support for proepared statements
* Connection Pool: each MySQL_Thread now has a pair thread, therefore the number of threads executed are mysql-threads x 2 . The second set of MySQL_Threads are responsible to only handle idle connections.
It also introduced two new global variables and a new global status:
* `mysql-session_idle_ms` (default 1000) : when a session is idle for that long, it is moved to a thread responsible to handle idle connections
* `mysql-session_idle_show_processlist` (default false) : specifies if idle connections are displayed on SHOW PROCESSLIST or on any query against `stats_mysql_processlist`
* `Client_Connections_non_idle` : returns the number of client connections that are not idle, therefore handled by the main MySQL_Threads and not moved to the second set of MySQL_Threads responsible to only handle idle connections
* Network: add support for IPv6 [#726](../../../../issues/726) and [#460](../../../../issues/460), thanks to @ton31337
* Connection Pool: support for SO_REUSEPORT, it can be only enabled on the command line with -r or --reuseport at startup
* Query Processor: added new variable mysql-digests_lowercase to always set digest to lowercase [#725](../../../../issues/725)
### Bug fixes
* Query Processor: rules with only digest were matching everything [#717](../../../../issues/717)
* Query Processor: rules without digest where incorrectly displayed [#719](../../../../issues/719)
* General: Unix Socket Domain file was not removed at shutdown [#714](../../../../issues/714)
* MySQL Protocol: upgrade from mariadb-connector-c 2.1.0 to 2.3.1 due to several bugs
* Connection Pool: server is not shunned if max_user_connections is reached for user [#737](../../../../issues/737)
## ProxySQL v1.3.0f
Release date: 2016-10-29
Compared to v1.3.0:
### New features
* killed query because of query_timeout returns error 1907 [#750](../../../../issues/750)
* removes any lower limit for query_timeout (before was 300ms)
* added support for `STMT_SEND_LONG_DATA` [#764](../../../../issues/764)
* new customer error codes for `Max connect timeout` [#761](../../../../issues/761) and [wiki](https://github.com/sysown/proxysql/wiki/Error-codes)
### Bug fixes
* `mysql_query_rules`.`log` not working [#723](../../../../issues/723)
* clients are disconnected if server is running for less than 8 hours [#744](../../../../issues/744)
* remove annoying error from Admin when running `\s` [#745](../../../../issues/745)
* escaping errors on `INSERT INTO stats_mysql_processlist` [#746](../../../../issues/746)
* kill threads were using wrong credentials due to encrypted passwords
* on slow network, client connections could become stall
* `session_idle_ms` was processed as us instead of ms
* `mysql_close` was trying to close prepared statements on already closed connection [#765](../../../../issues/765)
## ProxySQL v1.3.0g
Release date: 2016-11-14
Compared to v1.3.0f
### New features
* when a connection attempts receive error 1226, retry mechanism is disabled [#786](../../../../issues/786)
* better support for IPv6
* more verbose message on connection timeout [#776](../../../../issues/776)
* removed some race conditions during `PROXYSQL RESTART`
* `Access denied` message reports source (client) [#795](../../../../issues/795)
* `SHOW TABLES FROM xxx` is now sorted [#788](../../../../issues/788)
* allows rules on proxy_port without proxy_addr [#712](../../../../issues/712)
* Monitor threads are started without arena cache
* Monitor threads are started with 64KB stack
### Bug fixes
* validate `mysql` state before calling `mysql_stmt_free_result()` [#779](../../../../issues/779)
* `SHUNNED_REPLICATION_LAG` does not stop queries [#774](../../../../issues/774)
* `mysql_servers`.`comment` not loaded at runtime [#787](../../../../issues/787)
* fixed two memory leak in Monitor , reported in [#766](../../../../issues/766) and [#796](../../../../issues/796)
### Others
* upgraded jemalloc from 4.2.1 to 4.3.1
* jemalloc is compiled with `--enable-prof`
* added few memory sanitizers

@ -7,7 +7,7 @@
Summary: A high-performance MySQL proxy
Name: proxysql
Version: 1.3.0f
Version: 1.3.0h
Release: 1
License: GPL+
Group: Development/Tools

@ -7,7 +7,7 @@
Summary: A high-performance MySQL proxy
Name: proxysql
Version: 1.3.0f
Version: 1.3.0h
Release: 1
License: GPL+
Group: Development/Tools

@ -4,7 +4,7 @@ Homepage: http://www.proxysql.com
Standards-Version: 3.9.2
Package: proxysql
Version: 1.3.0f
Version: 1.3.0h
Maintainer: Rene Cannao <rene.cannao@gmail.com>
Architecture: amd64
# Changelog: CHANGELOG.md

@ -4,7 +4,7 @@ Homepage: http://www.proxysql.com
Standards-Version: 3.9.2
Package: proxysql
Version: 1.3.0f
Version: 1.3.0h
Maintainer: Rene Cannao <rene.cannao@gmail.com>
Architecture: amd64
# Changelog: CHANGELOG.md

@ -7,7 +7,7 @@
Summary: A high-performance MySQL proxy
Name: proxysql
Version: 1.3.0f
Version: 1.3.0h
Release: 1
License: GPL+
Group: Development/Tools

@ -4,7 +4,7 @@ Homepage: http://www.proxysql.com
Standards-Version: 3.9.2
Package: proxysql
Version: 1.3.0f
Version: 1.3.0h
Maintainer: Rene Cannao <rene.cannao@gmail.com>
Architecture: amd64
# Changelog: CHANGELOG.md

@ -4,7 +4,7 @@ Homepage: http://www.proxysql.com
Standards-Version: 3.9.2
Package: proxysql
Version: 1.3.0f
Version: 1.3.0h
Maintainer: Rene Cannao <rene.cannao@gmail.com>
Architecture: amd64
# Changelog: CHANGELOG.md

@ -4,7 +4,7 @@ Homepage: http://www.proxysql.com
Standards-Version: 3.9.2
Package: proxysql
Version: 1.3.0f
Version: 1.3.0h
Maintainer: Rene Cannao <rene.cannao@gmail.com>
Architecture: amd64
# Changelog: CHANGELOG.md

@ -228,8 +228,8 @@ class MySQL_STMT_Manager {
MySQL_STMT_Manager();
~MySQL_STMT_Manager();
int ref_count(uint32_t statement_id, int cnt, bool lock, bool is_client);
MySQL_STMT_Global_info * add_prepared_statement(unsigned int h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, bool lock=true);
MySQL_STMT_Global_info * add_prepared_statement(unsigned int h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, int _cache_ttl, int _timeout, int _delay, bool lock=true);
MySQL_STMT_Global_info * add_prepared_statement(bool *is_new, unsigned int h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, bool lock=true);
MySQL_STMT_Global_info * add_prepared_statement(bool *is_new, unsigned int h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, int _cache_ttl, int _timeout, int _delay, bool lock=true);
MySQL_STMT_Global_info * find_prepared_statement_by_stmt_id(uint32_t id, bool lock=true);
MySQL_STMT_Global_info * find_prepared_statement_by_hash(uint64_t hash, bool lock=true);
uint32_t total_prepared_statements() { return next_statement_id-1; }

@ -68,6 +68,7 @@ class MySQL_Connection {
MySrvC *parent;
MySQL_Connection_userinfo *userinfo;
MySQL_Data_Stream *myds;
enum MySerStatus server_status; // this to solve a side effect of #774
unsigned long largest_query_length;
uint32_t status_flags;
int async_exit_status; // exit status of MariaDB Client Library Non blocking API

@ -615,9 +615,9 @@ void * monitor_read_only_thread(void *arg) {
while (mmsd->async_exit_status) {
mmsd->async_exit_status=wait_for_mysql(mmsd->mysql, mmsd->async_exit_status);
unsigned long long now=monotonic_time();
if (now > mmsd->t1 + mysql_thread___monitor_ping_timeout * 1000) {
if (now > mmsd->t1 + mysql_thread___monitor_read_only_timeout * 1000) {
mmsd->mysql_error_msg=strdup("timeout check");
proxy_error("Timeout on read_only check for %s:%d. If the server is overload, increase mysql-monitor_read_only_timeout. Assuming read_only=1\n", mmsd->hostname, mmsd->port);
proxy_error("Timeout on read_only check for %s:%d after %lldms. If the server is overload, increase mysql-monitor_read_only_timeout. Assuming read_only=1\n", mmsd->hostname, mmsd->port, now-mmsd->t1);
goto __exit_monitor_read_only_thread;
}
if (GloMyMon->shutdown==true) {
@ -631,9 +631,9 @@ void * monitor_read_only_thread(void *arg) {
while (mmsd->async_exit_status) {
mmsd->async_exit_status=wait_for_mysql(mmsd->mysql, mmsd->async_exit_status);
unsigned long long now=monotonic_time();
if (now > mmsd->t1 + mysql_thread___monitor_ping_timeout * 1000) {
if (now > mmsd->t1 + mysql_thread___monitor_read_only_timeout * 1000) {
mmsd->mysql_error_msg=strdup("timeout check");
proxy_error("Timeout on read_only check for %s:%d. If the server is overload, increase mysql-monitor_read_only_timeout. Assuming read_only=1\n", mmsd->hostname, mmsd->port);
proxy_error("Timeout on read_only check for %s:%d after %lldms. If the server is overload, increase mysql-monitor_read_only_timeout. Assuming read_only=1\n", mmsd->hostname, mmsd->port, now-mmsd->t1);
goto __exit_monitor_read_only_thread;
}
if (GloMyMon->shutdown==true) {
@ -979,7 +979,7 @@ __end_monitor_connect_loop:
mysql_thread___monitor_history = mysql_thread___monitor_ping_interval * (mysql_thread___monitor_ping_max_failures + 1 );
}
unsigned long long time_now=realtime_time();
rc=sqlite3_bind_int64(statement, 1, time_now-mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
rc=sqlite3_bind_int64(statement, 1, time_now-(unsigned long long)mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
SAFE_SQLITE3_STEP(statement);
rc=sqlite3_clear_bindings(statement); assert(rc==SQLITE_OK);
rc=sqlite3_reset(statement); assert(rc==SQLITE_OK);
@ -1087,7 +1087,7 @@ __end_monitor_ping_loop:
mysql_thread___monitor_history = mysql_thread___monitor_ping_interval * (mysql_thread___monitor_ping_max_failures + 1 );
}
unsigned long long time_now=realtime_time();
rc=sqlite3_bind_int64(statement, 1, time_now-mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
rc=sqlite3_bind_int64(statement, 1, time_now-(unsigned long long)mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
SAFE_SQLITE3_STEP(statement);
rc=sqlite3_clear_bindings(statement); assert(rc==SQLITE_OK);
rc=sqlite3_reset(statement); assert(rc==SQLITE_OK);
@ -1312,7 +1312,7 @@ __end_monitor_read_only_loop:
mysql_thread___monitor_history = mysql_thread___monitor_ping_interval * (mysql_thread___monitor_ping_max_failures + 1 );
}
unsigned long long time_now=realtime_time();
rc=sqlite3_bind_int64(statement, 1, time_now-mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
rc=sqlite3_bind_int64(statement, 1, time_now-(unsigned long long)mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
SAFE_SQLITE3_STEP(statement);
rc=sqlite3_clear_bindings(statement); assert(rc==SQLITE_OK);
rc=sqlite3_reset(statement); assert(rc==SQLITE_OK);
@ -1426,7 +1426,7 @@ __end_monitor_replication_lag_loop:
mysql_thread___monitor_history = mysql_thread___monitor_ping_interval * (mysql_thread___monitor_ping_max_failures + 1 );
}
unsigned long long time_now=realtime_time();
rc=sqlite3_bind_int64(statement, 1, time_now-mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
rc=sqlite3_bind_int64(statement, 1, time_now-(unsigned long long)mysql_thread___monitor_history*1000); assert(rc==SQLITE_OK);
SAFE_SQLITE3_STEP(statement);
rc=sqlite3_clear_bindings(statement); assert(rc==SQLITE_OK);
rc=sqlite3_reset(statement); assert(rc==SQLITE_OK);

@ -161,8 +161,9 @@ int MySQL_STMT_Manager::ref_count(uint32_t statement_id, int cnt, bool lock, boo
if (s!=m.end()) {
MySQL_STMT_Global_info *a=s->second;
if (is_client) {
__sync_fetch_and_add(&a->ref_count_client,cnt);
ret=a->ref_count_client;
ret=__sync_add_and_fetch(&a->ref_count_client,cnt);
//__sync_fetch_and_add(&a->ref_count_client,cnt);
//ret=a->ref_count_client;
if (m.size() > (unsigned)mysql_thread___max_stmts_cache) {
int max_purge=m.size()/20; // purge up to 5%
int i=-1;
@ -170,7 +171,7 @@ int MySQL_STMT_Manager::ref_count(uint32_t statement_id, int cnt, bool lock, boo
for (std::map<uint32_t, MySQL_STMT_Global_info *>::iterator it=m.begin(); it!=m.end(); ++it) {
if (i==(max_purge-1)) continue;
MySQL_STMT_Global_info *a=it->second;
if (a->ref_count_client == 0) {
if (__sync_add_and_fetch(&a->ref_count_client,0) == 0) {
uint64_t hash=a->hash;
auto s2=h.find(hash);
if (s2!=h.end()) {
@ -206,11 +207,11 @@ int MySQL_STMT_Manager::ref_count(uint32_t statement_id, int cnt, bool lock, boo
return ret;
}
MySQL_STMT_Global_info * MySQL_STMT_Manager::add_prepared_statement(unsigned int _h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, bool lock) {
return add_prepared_statement(_h, u, s, q, ql, stmt, -1, -1, -1, lock);
MySQL_STMT_Global_info * MySQL_STMT_Manager::add_prepared_statement(bool *is_new, unsigned int _h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, bool lock) {
return add_prepared_statement(is_new, _h, u, s, q, ql, stmt, -1, -1, -1, lock);
}
MySQL_STMT_Global_info * MySQL_STMT_Manager::add_prepared_statement(unsigned int _h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, int _cache_ttl, int _timeout, int _delay, bool lock) {
MySQL_STMT_Global_info * MySQL_STMT_Manager::add_prepared_statement(bool *is_new, unsigned int _h, char *u, char *s, char *q, unsigned int ql, MYSQL_STMT *stmt, int _cache_ttl, int _timeout, int _delay, bool lock) {
MySQL_STMT_Global_info *ret=NULL;
uint64_t hash=stmt_compute_hash(_h, u, s, q, ql); // this identifies the prepared statement
if (lock) {
@ -223,6 +224,7 @@ MySQL_STMT_Global_info * MySQL_STMT_Manager::add_prepared_statement(unsigned int
//MySQL_STMT_Global_info *a=f->second;
//ret=a->statement_id;
ret=f->second;
*is_new=false;
} else {
// we need to create a new one
bool free_id_avail=false;
@ -246,6 +248,8 @@ MySQL_STMT_Global_info * MySQL_STMT_Manager::add_prepared_statement(unsigned int
ret=a;
//next_statement_id++; // increment it
//__sync_fetch_and_add(&ret->ref_count_client,1); // increase reference count
__sync_fetch_and_add(&ret->ref_count_client,1); // increase reference count
*is_new=true;
}
__sync_fetch_and_add(&add_prepared_statement_calls,1);
__sync_fetch_and_add(&ret->ref_count_server,1); // increase reference count

@ -1656,7 +1656,7 @@ __get_pkts_from_client:
mybe->server_myds->wait_until=0;
pause_until=0;
mybe->server_myds->killed_at=0;
//mybe->server_myds->mysql_real_query.init(&pkt);
mybe->server_myds->mysql_real_query.init(&pkt); // fix memory leak for PREPARE in prepared statements #796
client_myds->setDSS_STATE_QUERY_SENT_NET();
}
}
@ -2022,9 +2022,10 @@ handler_again:
case PROCESSING_STMT_PREPARE:
{
uint32_t stmid;
bool is_new;
MySQL_STMT_Global_info *stmt_info=NULL;
// if (mysql_thread___stmt_multiplexing) {
stmt_info=GloMyStmt->add_prepared_statement(current_hostgroup,
stmt_info=GloMyStmt->add_prepared_statement(&is_new, current_hostgroup,
(char *)client_myds->myconn->userinfo->username,
(char *)client_myds->myconn->userinfo->schemaname,
(char *)CurrentQuery.QueryPointer,
@ -2077,6 +2078,7 @@ handler_again:
} else {
client_myds->myprot.generate_STMT_PREPARE_RESPONSE(client_myds->pkt_sid+1,stmt_info);
client_myds->myconn->local_stmts->insert(stmt_info->statement_id,NULL);
if (is_new) __sync_fetch_and_sub(&stmt_info->ref_count_client,1);
}
}
break;
@ -2140,9 +2142,12 @@ handler_again:
}
*/
if (
(myconn->parent->status==MYSQL_SERVER_STATUS_OFFLINE_HARD) // the query failed because the server is offline hard
// due to #774 , we now read myconn->server_status instead of myconn->parent->status
(myconn->server_status==MYSQL_SERVER_STATUS_OFFLINE_HARD) // the query failed because the server is offline hard
||
(myconn->parent->status==MYSQL_SERVER_STATUS_SHUNNED && myconn->parent->shunned_automatic==true && myconn->parent->shunned_and_kill_all_connections==true) // the query failed because the server is shunned due to a serious failure
(myconn->server_status==MYSQL_SERVER_STATUS_SHUNNED && myconn->parent->shunned_automatic==true && myconn->parent->shunned_and_kill_all_connections==true) // the query failed because the server is shunned due to a serious failure
||
(myconn->server_status==MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG) // slave is lagging! see #774
) {
if (mysql_thread___connect_timeout_server_max) {
myds->max_connect_time=thread->curtime+mysql_thread___connect_timeout_server_max*1000;

@ -314,9 +314,9 @@ MySQL_Threads_Handler::MySQL_Threads_Handler() {
variables.monitor_connect_timeout=200;
variables.monitor_ping_interval=60000;
variables.monitor_ping_max_failures=3;
variables.monitor_ping_timeout=100;
variables.monitor_ping_timeout=1000;
variables.monitor_read_only_interval=1000;
variables.monitor_read_only_timeout=100;
variables.monitor_read_only_timeout=800;
variables.monitor_replication_lag_interval=10000;
variables.monitor_replication_lag_timeout=1000;
variables.monitor_query_interval=60000;
@ -1870,6 +1870,7 @@ bool MySQL_Thread::init() {
GloQPro->init_thread();
refresh_variables();
i=pipe(pipefd);
ioctl_FIONBIO(pipefd[1],1);
mypolls.add(POLLIN, pipefd[0], NULL, 0);
assert(i==0);
return true;
@ -2127,7 +2128,7 @@ __run_skip_1:
unsigned char c=1;
int fd=thr->pipefd[1];
if (write(fd,&c,1)==-1) {
proxy_error("Error while signaling maintenance thread\n");
//proxy_error("Error while signaling maintenance thread\n");
}
}
}
@ -2510,7 +2511,7 @@ __run_skip_2:
//MySQL_Thread *thr=GloMTH->mysql_threads[w].worker;
int fd=thr->pipefd[1];
if (write(fd,&c,1)==-1) {
proxy_error("Error while signaling maintenance thread\n");
//proxy_error("Error while signaling maintenance thread\n");
}
}
} else {
@ -2525,7 +2526,7 @@ __run_skip_2:
//MySQL_Thread *thr=GloMTH->mysql_threads[w].worker;
int fd=thr->pipefd[1];
if (write(fd,&c,1)==-1) {
proxy_error("Error while signaling maintenance thread\n");
//proxy_error("Error while signaling maintenance thread\n");
}
}
//spin_wrunlock(&GloMTH->rwlock_resumes);

@ -604,7 +604,7 @@ handler_again:
break;
case ASYNC_CONNECT_TIMEOUT:
//proxy_error("Connect timeout on %s:%d : %llu - %llu = %llu\n", parent->address, parent->port, myds->sess->thread->curtime , myds->wait_until, myds->sess->thread->curtime - myds->wait_until);
proxy_error("Connect timeout on %s:%d : exceeded by %lluus\n", myds->sess->thread->curtime - myds->wait_until);
proxy_error("Connect timeout on %s:%d : exceeded by %lluus\n", parent->address, parent->port, myds->sess->thread->curtime - myds->wait_until);
parent->connect_error(mysql_errno(mysql));
break;
case ASYNC_CHANGE_USER_START:
@ -1114,12 +1114,13 @@ int MySQL_Connection::async_query(short event, char *stmt, unsigned long length,
PROXY_TRACE();
assert(mysql);
assert(ret_mysql);
server_status=parent->status; // we copy it here to avoid race condition. The caller will see this
if (
(parent->status==MYSQL_SERVER_STATUS_OFFLINE_HARD) // the server is OFFLINE as specific by the user
(server_status==MYSQL_SERVER_STATUS_OFFLINE_HARD) // the server is OFFLINE as specific by the user
||
(parent->status==MYSQL_SERVER_STATUS_SHUNNED && parent->shunned_automatic==true && parent->shunned_and_kill_all_connections==true) // the server is SHUNNED due to a serious issue
(server_status==MYSQL_SERVER_STATUS_SHUNNED && parent->shunned_automatic==true && parent->shunned_and_kill_all_connections==true) // the server is SHUNNED due to a serious issue
||
(parent->status==MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG) // slave is lagging! see #774
(server_status==MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG) // slave is lagging! see #774
) {
return -1;
}

Loading…
Cancel
Save