Added new array mysql_tracked_variables[] that defines the tracked variables.
Ideally, tracking a new session variable will requires adding only a couple of lines of code.
Removed session variables from mysql_thread_variables_names:
- now both MySQL_Threads_Handler::get_variables_list() and MySQL_Threads_Handler::has_variable() rely on mysql_tracked_variables[]
Simplified MySQL_Variables, as it now also uses mysql_tracked_variables[] .
Simplify parser in MySQL_Session
Generalized code to handle two variables together:
- sql_auto_is_null
- sql_safe_updates
Migrating more session variables to new algorithm
- collation_connection
- net_write_timeout
- max_join_size
Migrating more session variables to new algorithm
- collation_connection
- net_write_timeout
- max_join_size
libsqlinjection generates a lot of false positives.
This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints
This table can list fingerprints generated by libsqlinjection:
if the fingerprint is listed in this table, proxysql will consider it as
a false positive.
This commit also enables SQLi algorithm only if the query is not already
explicitly whitelisted.
Library libjection is being used.
New variable mysql-automatic_detect_sqli defines if the feature is enabled or not.
If an SQL injection is detected:
- the client connection is immediately terminated
- an entry in the error log is reported
* introduced 2 new variables:
- mysql-firewall_whitelist_enabled : defines if the firewall with whitelist algorithm is enabled or not
- firewall_whitelist_errormsg : error message returned to the client if an error message is not already set
* Query_Processor returns firewall_whitelist_mode to the calling MySQL_Session, but the logging on error log currently happens within the Query_Processor itself
* if mysql-firewall_whitelist_enabled is turned on and a user it is not found in mysql_firewall_whitelist_users , the default behavior is that query is rejected
* lookup on mysql_firewall_whitelist_users happens for username@ip (specific IP) and username@'' (all IPs) . Match on network subnet is currently not implemented
* implemented SAVE MYSQL FIREWALL FROM RUNTIME
* fixed implementation of SAVE MYSQL FIREWALL TO DISK
* runtime configuration can be viewed in runtime_mysql_firewall_whitelist_rules and runtime_mysql_firewall_whitelist_users
This commit fixes the following bug:
if a client connection uses SSL and sends a query larger than 32KB, the query is never executed and the connection hang
Initial support for `SET SESSION TRANSACTION READ ONLY` or `READ WRITE`.
Extended `SET` parser to support also `SET SESSION TRANSACTION`.
Hostgroup Manager doesn't kill backend connections in case of error 1231.
`autocommit` is set at session level but also on MySQL client connection.
Added several debugging entries.
Several `handler_again___verify_backend_*` functions are disabled if `locked_on_hostgroup` is enabled.
Added 2 new status variables:
- queries_with_max_lag_ms__delayed
- queries_with_max_lag_ms__total_wait_time_us
Do not get replication lag from replicas if the value is 0
Fixed an error in the computation of max_lag_ms
This should fix a lot of issues related to failed parsing of SET statement.
This and the two 2 previous commits introduce several status variables, and a
new configuration variable: mysql-set_query_lock_on_hostgroup
Possible values for mysql-set_query_lock_on_hostgroup:
- 0 : legacy behavior , before 2.0.5
- 1 : (default) . SET statements that cannot be parsed correctly disable
both multiplexing AND routing. Attempting to route traffic while a
connection is linked to a specific backend connection will trigger
an error to be returned to the client
Issue #2120 : Send SESSION_TRACK_GTIDS to client
Issue #2121 : Track CLIENT_FOUND_ROWS required by the client
Issue #2125 : Track CLIENT_MULTI_STATEMENTS required by the client
Enhancements:
- added metrics rows_affected and rows_sent
- added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0
- added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871
Changing value at runtime causes the current file to be closed and a new one created
- fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged
Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent
Extended `eventslog_reader_sample.cpp` to support the new enhancements
MySQL_HostGroups_Manager::get_MyConn_from_pool() inserts connection
to mysrvc->ConnectionUsed, and this connections is also inserted
to My_Thread->cached_connections which leads to double deallocation.
Fixed https://github.com/sysown/proxysql/issues/1985.
After execute "PROXYSQL STOP", main() -> ProxySQL_Main_init_phase4___shutdown()
is called, then main() -> ProxySQL_Main_init_phase2___not_started() is called, this
function creates GloMTH and waits on GloVars.global.start_mutex, thus
ProxySQL_Main_init_phase3___start_all() -> ProxySQL_Main_init_MySQL_Threads_Handler_module()
isn't called.
Before "PROXYSQL START" is executed, admin module will crash on query to stats_mysql_processlist.
Similar cause to https://github.com/sysown/proxysql/issues/2037 and
https://github.com/sysown/proxysql/issues/2065.
Added new command `PROXYSQL INTERNAL SESSION` that clients can execute to
receive internal information about their own connection in JSON format.
Added JSON library.
Recompiled SQLite3 to support JSON.
Added new column `extended_info` in `stats_mysql_processlist`.
Added new mysql variable `mysql-show_processlist_extended` that determine the
content of `stats_mysql_processlist.extended_info`:
- 0 : no info
- 1 : JSON format
- 2 : JSON format with pretty printing
aws_aurora_replicas_skipped_during_query is a status variable for better
monitoring behavior due to replication lag in AWS Aurora
Also fixed lag computing in connection pool
Epoll thread does not poll for EPOLLOUT. If a throttled session with
pending data out is moved to epoll thread, the session will stay in
epoll thread indefinitely, causing a hang observed by client. Fix by not
moving throttled session to epoll thread.
Fix#1939
Temporary disable multiplexing when last_insert_id is returned in OK packet.
Multiplexing is disabled for mysql-auto_increment_delay_multiplex queries.
mysql-auto_increment_delay_multiplex ranges from 0 to 1000000 .
Default value is 5
When shunning a node, evaluate the different between
`mysql-monitor_ping_interval` and `mysql-shun_recovery_time_sec`, and postpone
in the future when the server needs to be recovered
- if clients uses mysql_native_password, if LDAP is enabled and if the user doesn't exist, switch to mysql_clear_password.
- if neither mysql_native_password or mysql_clear_password are used by the client:
- if LDAP is not enabled, always switch to mysql_native_password
- if LDAP is enabled:
- if the user exists, switch to mysql_native_password
- if the user doesn't exists, switch to mysql_clear_password
Added MySQL variable mysql-add_ldap_user_comment to determine if a comment with the original username needs to be added in the queries.
This commit also tracks the charset during the first handshake response.
Both KILL QUERY and KILL CONNECTION work
The only security check enforced is that the user sending the KILL
is the same user of the connection/query being killed.
The same applies also for:
- runtime_mysql_replication_hostgroups
- runtime_mysql_group_replication_hostgroups
- runtime_mysql_galera_hostgroups
Close#1435 and #1436
On a commodity hardware, performance improved from 25us to 7us
Performance can be improved more if replacing legacy algorithm of searching variables by name.
Although the legacy algorithm allow to add plugins in future.
Added 10 new metrics
Added 3 new global variables
* monitor_threads_min : minimum number of threads
* monitor_threads_max : maximum number of threads
* monitor_threads_queue_maxsize : maximum numbr of pending checks before starting new threads
Variable reset_connection_algorithm could either be:
1 = algorithm used up too version 1.4
2 = algorithm new since ProxySQL 2.0 (now default)
When reset_connection_algorithm = 2 , MySQL_Thread itself tries to reset connections instead of relying on connections purger HGCU_thread_run()
Statistics collected aboout GTID queries
Statistics displayed on HTTP server
Count number of GTID events per server
Online upgrade of all mysql_connections tables
Fixed path for libev
This commits adds a new variable: mysql-monitor_replication_lag_use_percona_heartbeat
This variable defines the percona heartbeat table used to check replication lag.
If set, replication lag is checked against the defined table, otherwise `SHOW SLAVE STATUS` is used.
To be set, the value should match the following regex:
```
`?([a-z\d_]+)`?\.`?([a-z\d_]+)`?
```
If variable mysql-verbose_query_error is set, "Error during query" message will be extended adding:
- username
- client IP
- schemaname
- digest of the original query (not the original query itself)
If autocommit_false_is_transaction=true (false by default), a connection
with autocommit=0 is treated as a transaction.
If forward_autocommit=true (false by default), the same behavior applies.
Variable name is `mysql-throttle_connections_per_sec_to_hostgroup` .
Curently it is a global variable and limits the number of new connections per
hostgroup, and not per specific node.
For example, if mysql-throttle_connections_per_sec_to_hostgroup=100, no more
than 100 new connections can be created on any hostgroup no matter the number
of servers in that hostgroup.
The default is very high (1000000) thus not changing default behaviour.
Tuning this variable allows to control and throttle connections spikes to the
backend servers.
Added also new status variable `Server_Connections_delayed`.
This is a counter of how many times Hostgroup Manager didn't return a
connection because the limit was reached. It is worth to note that a single
client request could make multiple requests, therefore this variable counts
the number of time a new connection wasn't created and not how many requests
were delayed.
Introduced 2 new global variables:
* mysql-stats_time_backend_query (default true)
* mysql-stats_time_query_processor (default true)
For backward compatibility, they are both enabled by default
Automatically reset mysql->insert_id when a connection is sent to connection pool.
If is not reset to 0 , when a backend connections is assigned to a client, this one can potentially read the last inserted id from the previous client.
Enforced max_writers
Enforced writer_is_also_reader
Increased the number of entries in mysql_server_group_replication_log to 100 per node
Lowered the minimum value for mysql-monitor_groupreplication_healthcheck_interval to 50ms
Lowered the minimum value for mysql-monitor_groupreplication_healthcheck_timeout to 50ms
CentOS 5 doesn't have support for epoll_create1(), so I've defined a
macro to fall back on epoll_create() in that case.
Note that the `1` argument to epoll_create() is actually ignored, and is
only there to conform to the requirement that it be larger than 0.
The 2 new variables are:
* mysql-query_digests_max_digest_length : defines the maximum length of digest_text as reported in stats_mysql_query_digest
* mysql-query_digests_max_query_length : defines the maximum query length processed when computing query's digest and digext_text
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
This commit is able to speed up LOAD MYSQL SERVERS TO RUNTIME up to 10x for very large mysql_servers table with tens of thousands of servers.
Main improvements:
* removed 1 unnecessary call to `generate_mysql_servers_table()`
* replaced 1 call to `generate_mysql_servers_table()` with UPDATE statements when necessary
* implemented new function `servers_add()` that replaces `server_add()` to perform bulk operation
* replaced calls to `sqlite3_exec()` with proper parameters bindings
* use of multirows INSERT statements (32 rows at the time)
Miror improvements:
* added new variable mysql-hostgroup_manager_verbose , 1 by default for backward compatibility and debugging
* in not debug mode, access to HGM db has no shared cache
* in not debug mode, tables in HGM have checks disabled
Further note:
* Once issue #797 is completed, `mysql_servers` in MyHGM can be simplified further
Increased mysql-monitor_ping_timeout from 100 to 1000 milliseconds.
Also increased monitor_read_only_timeout from 100 to 800 milliseconds as read_only timeout is even more important than ping timeout.
Metrics introduced:
* ConnPool_get_conn_failure : connection pool cannot provide any connection
* ConnPool_get_conn_immediate : connection is provided from per-thread cache
* ConnPool_get_conn_success : the session is able to get a connection, either from per-thread cache or connection pool
MySQL_STMTs_meta() is aware to which session it belongs
entries are removed from MySQL_STMTs_meta() when unused
find_prepared_statement_by_hash() increases the client ref count
Hardcoded a maximum number of statements
The called of MySQL_STMTs_local functions doesn't must specify if it is a client or a backend.
Added also a function in MySQL_Connection to make the connection aware it is a client.
When query rules are processed, even if flagOUT is set rules are processed
in numerical order.
This new variable allows to jump back (reset rule_id to 0) N amount of times.
Using this variable is possible to implement loops.
Futhermore, a loop is possible with only one rule setting flagIN=flagOUT .
The default is 0. The maximum value is 1000000 .
This variable limits the amount of data that the client can send to the backend.
Note that ProxySQL won't really limit the amount of data that the client can send to the proxy, but the amount of data that the prox can send to the backend.
This commits also fixes issue #526Closes#526
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
Add new variable MySQL_Session::mirror (false by default)
Add new struct MySQL_Session::mirrorPkt (false by default)
These functions return immediately if this is a mirror session:
* MySQL_Protocol::generate_pkt_EOF()
* MySQL_Protocol::generate_pkt_ERR()
* MySQL_Protocol::generate_pkt_OK()
* MySQL_Protocol::generate_pkt_column_count()
* MySQL_Protocol::generate_pkt_field()
* MySQL_Protocol::generate_pkt_row3()
MySQL_ResultSet::MySQL_ResultSet() exits almost immediately if this is a mirror session
Most of the code of these functions is not executed if this is a mirror session:
* MySQL_ResultSet::add_eof()
* MySQL_ResultSet::get_resultset()
In MySQL_Session::writeout() :
* `client_myds->write_to_net_poll()` is called only if this is not a mirror session
In MySQL_Session::handler():
* if query is a `SELECT` (hardcoded for now) :
** create a new session and mark it as `mirror=true`
** duplicate the query and send it to the new session
* if `mirror==true` :
** pretend to read a new query from c`lient_myds`
** proceed normally (with the only exception that no data can be sent to a client)
In MySQL_Thread::process_all_sessions() :
* if `mirror==true` and `status==WAITING_CLIENT_DATA` it means the session is "completed" so it get destroyed
Limitations:
* it currently doesn't support large packets
* doesn't support different charset
* doesn't support transactions