Add initial support for FR limiting the number of connection errors that
can be initiated from a particular address before deniying future
connections from that address. Feature is controlled by new introduced
variables:
- mysql-client_host_error_counts
- mysql-client_host_cache_size
Command PROXYSQL RELOAD TLS is able to load new key/cert files and
create a new SSL context.
Loading of SSL can happens in two stages:
* during bootstrap: if it fails, proxysql dies
* running PROXYSQL RELOAD TLS: if it fails, proxysql will keep the old SSL context
This new method should successfully handle errors like missing or corrupted files.
We are also adding two buffers used to store the key/cert, to be used by the web interface.
This commit add the support of ERR packet in binary protocol resultset,
for example due to MAX_EXECUTION_TIME while running a prepared statement.
It also changes how EOF packets are added: instead of writing the EOF in the
resultset buffer, the EOF packet is written in its own packet. This allows to
easily remove it if needed.
It also fix what seems (not sure) a bug in MariaDB client library, where if a
prepared statement is interrupted it cannot be executed again
Added function MySQL_Session::detected_broken_connection() that unifies the way broken connections are logged.
It also logs username, and last time used.
Added also a new macro proxy_error_inline()
'CurrentQuery.QueryParserArgs.first_comment' needs to hold it's own copy
of 'first_comment', otherwise, the 'first_comment' from global 'stmt_info'
will be freed by 'Query_Info::end' at the end of the life of 'CurrentQuery'.
All variables that are not known by ProxySQL should fail to be parsed.
A best effor should also be done for tracking all the variables present
in a 'SET' statement prior to returning the parsing failure.
If a backend connection was terminated in the middle of the execution of multiple
statements in a multi-statements command, the worker thread was entering an infinite loop.
Furthermore, this commit disables query retry in case of multi-statements if the first one
completed successfully
This fixes#3339
This commit is the port to 2.1 of 90a4922ed8
* MySQL_ResultSet() stores a pointer to the PS
* MySQL_ResultSet::init_with_stmt() doesn't need anymore the pointer to the PS
Furthermore, it isn't anymore an "init" but an end (we will rename it)
* Added MySQL_ResultSet::add_row(MYSQL_ROWS *rows) for PS only
* MySQL_Protocol::generate_pkt_row3() accepts an optional row length.
If passed, it will know the length of the row without computing it
* MyRS initialized during ASYNC_STMT_EXECUTE_STORE_RESULT_START
* Implemented throttling during ASYNC_STMT_EXECUTE_STORE_RESULT_START
* MySQL_Connection::process_rows_in_ASYNC_STMT_EXECUTE_STORE_RESULT_CONT() always
generates a heartbeat. For now unnecessary, and we will better tune it later
* test_ps_async-t.cpp tunes mysql-threshold_resultset_size to trigger buffering
More testing is required
Few special queries are handled separately in ProxySQL
1. select @@version_comment limit 1
2. SHOW WARNINGS
3. SELECT USER()
4. PROXYSQL INTERNAL SESSION
5. SELECT LAST_INSERT_ID()
6. SELECT CONNECTION_ID()
7. C code mysql_set_server_option(mysql, MYSQL_OPTION_MULTI_STATEMENTS_ON)
We now remove EOF1 for these and respond with OK packets intead of EOF2
Former max_transaction_time is now renamed to max_transaction_idle_time.
mysql-max_transaction_idle_time defines the maximum time a transaction can stay idle.
mysql-max_transaction_time defines the maximum time of a transaction since its start.
In handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_qpo()
Changing the default of exit_after_SetParse from false to true.
If a variable isn't parsed or it needs to be forwarded, exit_after_SetParse is set to false
Changing the mapping logic in SESSION_TRACK_GTIDS in frontend and backend connections.
Up to now, frontend and backend `SESSION_TRACK_GTIDSs match.
This is now changed:
* backend connections are by default set to `mysql-default_session_track_gtids`
* if `mysql-default_session_track_gtids=OFF` (the default) , `session_track_gtids` is not changed on backend
* if the client asks for `session_track_gtids=OFF` , proxysql ignores it (it just acknowledge it)
* if the client asks for `session_track_gtids=OWN_GTID` , proxysql will apply it
* if the client asks for `session_track_gtids=ALL_GTIDS` , proxysql will switch to OWN_GTID and generate a warning
* if the backend doesn't support `session_track_gtids` (for example in MySQL 5.5 and MySQL 5.6), proxysql won't apply it. It knows checking server capabilities
This commit also deprecates function `MySQL_Session::handler_again___verify_backend__generic_variable()`
ProxySQL is now also able to know when proxysql binlog reader is perhaps in use tracking all gtid_port in mysql_servers.
This feature in future will be used to deprecate `mysql-default_session_track_gtids`
- Added extra doc about the 'status_flags' and 'server_status'.
- Added 'sql_log_bin0' and 'prepared_statement' to the 'conn.status'
from PROXYSQL SESSION INFO.
When a backend connection is reused for a different user, the data stream status is set to `STATE_MARIADB_QUERY`, but was not set back to `STATE_MARIADB_GENERIC` once the user was switched successfully.
This caused additional connection setup (like executing `init_connect`) to be skipped on the first query after switching users. If another query using the the same user was executed, this new query would end up performing this additional connection setup.
By setting the data stream status back to `STATE_MARIADB_GENERIC`, we can ensure connection setup happens correctly after switching users.
When a new backend connection was selected, the `session_track_gtids` config variable would be marked as set, but was never actually set on the backed connection.
This commit prepares proxysql for further development related to SQLite3.
Specifically, it allows to either use built-in SQLite3 library, or to load it from a plugin.
Adding the following methods:
* handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_STMT_RESET()
* handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_STMT_CLOSE()
* handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_STMT_SEND_LONG_DATA()
The two main blocks are:
* get_pkts_from_client() : handling the main loop around label __get_pkts_from_client
* handler() : handling the main loop around label handler_again
First commit to simplify MySQL_Session::handler()
It handles the following statuses:
* CHANGING_USER_SERVER
* CHANGING_AUTOCOMMIT
* SETTING_MULTI_STMT
* SETTING_SESSION_TRACK_GTIDS
* SETTING_SET_NAMES
From 1.3.0 till 2.0.x , when COM_STMT_PREPARE is executed and Query Processor is executed, metadata are stored in MySQL_STMT_Global_info().
These metadata include routing information.
Is a prepared statement is prepared on a specific hostgroup, it will always be executed on the same hostgroup.
This causes negative consequences, like the one reported in issue #2691 .
This commit:
* remove hostgroup_id from MySQL_STMT_Global_info() and other metadata
* Query Processor is called also during COM_STMT_EXECUTE
* removes hostgroup from table stats_mysql_prepared_statements_info
* Query Processor do not perform any query rewrite ruding COM_STMT_EXECUTE
* removed 14 statuses
* configured the MySQL_Session status to SETTING_VARIABLE for 14 session variables
* extended MySQL_Session to introduce a new variable that tracks the session variable to change
* MySQL_Threads_Handler::SQL3_Processlist() can handle all variables
* removed unnecessary loops
* improved performance removing unnecessary function calls
Added new array mysql_tracked_variables[] that defines the tracked variables.
Ideally, tracking a new session variable will requires adding only a couple of lines of code.
Removed session variables from mysql_thread_variables_names:
- now both MySQL_Threads_Handler::get_variables_list() and MySQL_Threads_Handler::has_variable() rely on mysql_tracked_variables[]
Simplified MySQL_Variables, as it now also uses mysql_tracked_variables[] .
Simplify parser in MySQL_Session
Generalized code to handle two variables together:
- sql_auto_is_null
- sql_safe_updates
Migrating more session variables to new algorithm
- collation_connection
- net_write_timeout
- max_join_size
Migrating more session variables to new algorithm
- collation_connection
- net_write_timeout
- max_join_size
Added new array mysql_tracked_variables[] that defines the tracked variables.
Ideally, tracking a new session variable will requires adding only a couple of lines of code.
Removed session variables from mysql_thread_variables_names:
- now both MySQL_Threads_Handler::get_variables_list() and MySQL_Threads_Handler::has_variable() rely on mysql_tracked_variables[]
Simplified MySQL_Variables, as it now also uses mysql_tracked_variables[] .
Simplify parser in MySQL_Session
Generalized code to handle two variables together:
- sql_auto_is_null
- sql_safe_updates
Migrating more session variables to new algorithm
- collation_connection
- net_write_timeout
- max_join_size
Migrating more session variables to new algorithm
- collation_connection
- net_write_timeout
- max_join_size
* Query_Processor::find_firewall_whitelist_rule() wasn't correctly finding rules
* SQL injection algorithm should only be triggered if the query isn't blocked already
libsqlinjection generates a lot of false positives.
This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints
This table can list fingerprints generated by libsqlinjection:
if the fingerprint is listed in this table, proxysql will consider it as
a false positive.
This commit also enables SQLi algorithm only if the query is not already
explicitly whitelisted.
This commit fix the following bug:
If some session variables were set by the client before proxysql turned on
lock_hostgroup, these variables were not set.
This commit also makes proxysql aware that this query causes lock_hostgroup:
`SET SESSION information_schema_stats_expiry=0`
Library libjection is being used.
New variable mysql-automatic_detect_sqli defines if the feature is enabled or not.
If an SQL injection is detected:
- the client connection is immediately terminated
- an entry in the error log is reported
- ability to handle transactions
- in OK packets and resultsets, SERVER_STATUS_IN_TRANS flag is set if a transaction is running
- a persistent database file in datadir is used instead of in-memory database
- database file uses WAL (Write-Ahead Logging)
- WAL journal size is set to 64MB
- monitor user can always login to SQLite3 Server
- in case of database lock within a transaction, an error is immediately returned to client
It a SET statement wasn't parsed correctly, hostgroup was locked
on current hostgroup and not on the hostgroup specified in
mysql_query_rules.destination_hostgroup
Initial support for `SET SESSION TRANSACTION READ ONLY` or `READ WRITE`.
Extended `SET` parser to support also `SET SESSION TRANSACTION`.
Hostgroup Manager doesn't kill backend connections in case of error 1231.
`autocommit` is set at session level but also on MySQL client connection.
Added several debugging entries.
Several `handler_again___verify_backend_*` functions are disabled if `locked_on_hostgroup` is enabled.
- switched from SSLv23_server_method() to TLS_server_method
- enforced min proto version to TLS1_VERSION using SSL_CTX_set_min_proto_version()
- when inspecting session status, cipher used is exported
Added 2 new status variables:
- queries_with_max_lag_ms__delayed
- queries_with_max_lag_ms__total_wait_time_us
Do not get replication lag from replicas if the value is 0
Fixed an error in the computation of max_lag_ms
This should fix a lot of issues related to failed parsing of SET statement.
This and the two 2 previous commits introduce several status variables, and a
new configuration variable: mysql-set_query_lock_on_hostgroup
Possible values for mysql-set_query_lock_on_hostgroup:
- 0 : legacy behavior , before 2.0.5
- 1 : (default) . SET statements that cannot be parsed correctly disable
both multiplexing AND routing. Attempting to route traffic while a
connection is linked to a specific backend connection will trigger
an error to be returned to the client
Issue #2120 : Send SESSION_TRACK_GTIDS to client
Issue #2121 : Track CLIENT_FOUND_ROWS required by the client
Issue #2125 : Track CLIENT_MULTI_STATEMENTS required by the client
Enhancements:
- added metrics rows_affected and rows_sent
- added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0
- added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871
Changing value at runtime causes the current file to be closed and a new one created
- fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged
Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent
Extended `eventslog_reader_sample.cpp` to support the new enhancements
In `stats_mysql_free_connections`, exported:
* address of connection
* address of mysql struct
* mysql thread_id
In `stats_mysql_processlist`, exported:
* address of session
* address of data stream
* address of connection
* address of mysql struct
* mysql thread_id
* removed password from non-debug built
Added new command `PROXYSQL INTERNAL SESSION` that clients can execute to
receive internal information about their own connection in JSON format.
Added JSON library.
Recompiled SQLite3 to support JSON.
Added new column `extended_info` in `stats_mysql_processlist`.
Added new mysql variable `mysql-show_processlist_extended` that determine the
content of `stats_mysql_processlist.extended_info`:
- 0 : no info
- 1 : JSON format
- 2 : JSON format with pretty printing
`SERVER_STATUS_NO_BACKSLASH_ESCAPES` is now tracked.
`set sql_mode` is executed immediately if client executes `set sql_mode`
specifying `NO_BACKSLASH_ESCAPES`.
A backend connection with `SERVER_STATUS_NO_BACKSLASH_ESCAPES` enabled has multiplexing immediately disabled.
Temporary disable multiplexing when last_insert_id is returned in OK packet.
Multiplexing is disabled for mysql-auto_increment_delay_multiplex queries.
mysql-auto_increment_delay_multiplex ranges from 0 to 1000000 .
Default value is 5
- if clients uses mysql_native_password, if LDAP is enabled and if the user doesn't exist, switch to mysql_clear_password.
- if neither mysql_native_password or mysql_clear_password are used by the client:
- if LDAP is not enabled, always switch to mysql_native_password
- if LDAP is enabled:
- if the user exists, switch to mysql_native_password
- if the user doesn't exists, switch to mysql_clear_password
Added MySQL variable mysql-add_ldap_user_comment to determine if a comment with the original username needs to be added in the queries.
This commit also tracks the charset during the first handshake response.
If:
* client wants autocommit=0
* enforce_autocommit_on_reads=false
* there is no transaction
* this seems to be the first query, and a SELECT not FOR UPDATE
Action:
* switch back to autcommit=1