Added function MySQL_Session::detected_broken_connection() that unifies the way broken connections are logged.
It also logs username, and last time used.
Added also a new macro proxy_error_inline()
See issue #3465
If a user (configurable) connects to proxysql without specifying a session transaction isolation level, proxysql will automatically assigns one to that user connection. Of course, said user can override such default session transaction isolation level running `SET SESSION TRANSACTION ISOLATION LEVEL` on a per-connection basis.
This feature can be configured extending `mysql_users.attributes` , adding an object named `default-transaction_isolation` .
An example of configuration could be:
```
UPDATE mysql_users SET attributes='{"default-transaction_isolation":"READ COMMITTED"}' WHERE username='sbtest';
```
This feature also requires input validation making sure that only valid isolation levels are used.
Furthermore, both dash and space can be used.
If the isolation level specified is not valid, it should be automatically removed from runtime.
'CurrentQuery.QueryParserArgs.first_comment' needs to hold it's own copy
of 'first_comment', otherwise, the 'first_comment' from global 'stmt_info'
will be freed by 'Query_Info::end' at the end of the life of 'CurrentQuery'.
All variables that are not known by ProxySQL should fail to be parsed.
A best effor should also be done for tracking all the variables present
in a 'SET' statement prior to returning the parsing failure.
1. Introduced extra paranethesis at the beggining and end of the regex for
maching queries surrounded by them like: `( SELECT ... )`.
2. Added explicit SELECT at the beggining to mach queries starting with
`SELECT`.
2. Introduced a new 'OR' in the first part of the regex: `|(?:NULL)` in
order to avoid collapsing NULL values followed by commas without spaces.
3. Improved final section of the regex `(?:)` that was previously
macthing with multiple empty groups, inducing a complete collapse of
the initial query into an empty value in case of not matching the initial
capturing group. With the new one, in case of emtpy value the external
parenthesis are matched, for being remove later from the value.
When a cluster node (client) connects to another cluster node, it
advertises its UUID but also exports its `admin-mysql_ifaces` .
The receiving cluster can use this information to try to guess how
to connect to the node that initialized the connection, for example
to collect metrics.
This is visible in table `stats_proxysql_servers_clients_status` .
HostGroups are stored in an array of pointers.
This commit allows the lookup using an unordered_map if the number of
Hostgroups is greater than 100.
For less number of hostgroups, a sequential scan seems more efficient.
This also introduce a test for micro benchmark.
This commit rewrites MySQL_HostGroups_Manager::read_only_action() in way to
reduce the calls to SQLite. It relies on two new std::set() :
- read_only_set1: if the server is already in a writer hostgroup
- read_only_set2: a cache to track the first time that a server has RO=0
Added also an index on MySQL_HostGroups_Manager,
ON mysql_servers (hostname,port)
If a backend connection was terminated in the middle of the execution of multiple
statements in a multi-statements command, the worker thread was entering an infinite loop.
Furthermore, this commit disables query retry in case of multi-statements if the first one
completed successfully
This fixes#3339
This commit is the port to 2.1 of 90a4922ed8
- Fixed return of proper http error codes on script failure or
internal failure to perform script execution. Introduced error
codes:
- 424: Script execution failed, either due to script failing to
be executed, exiting with non-zero error code or output
not being a valid JSON.
- 500: Internal error while trying to execute script.
- 200: Script was able to be executed properly, and result
was properly parsed.
- Fixed dangling process left by 'return' in child process
resulted from 'fork()' call.
- Fixed limitation of maximum return size of 48kb from executed
script.
- Improved usage of 'read()' by using NONBLOCKING pipes.
- Fixed leaving child process alive after timeout expiration, now
process is properly terminated.
- Improved general logging and returned JSONs in case of errors.
ProxySQL will now automatically generate a UUID during start up.
The UUID can be either passed through the command line using -U or --uuid,
or saved on disk in a new table named global_settings.
The new table is used to distinguish it from global_variables that is replicated.
Table global_settings isn't replicated by Cluster.
If UUID is not present, it is automatically generated and saved.
When processing COM_CHANGE_USER from a client, we did not parse the charset/collation.
This commit parses and processes also charset in COM_CHANGE_USER
We call mysql_options with MYSQL_SET_CHARSET_NAME if collation ID >= 255 .
This to solve the problem of MySQL server returning an Access denied if the backend doesn't support a collation during mysql_change_user
In mysql_init() charset is set to NULL .
In mysql_read_connect() charset is not changed if already set
This allows proxysql to change it outside the library
`generate_pkt_field2` is a less generic version of `generate_pkt_field`
and it is meant to be used only for queries executed on backend.
The function takes a `MYSQL_FIELD` as one of its arguments, thus a lot of
lengths do not need to be computed
Servers have no reason to be present in the reader hostgroup for 'writer_is_also_reader=2'
when they are present in any other hostgroup that is not 'backup_writer'.
* MySQL_ResultSet() stores a pointer to the PS
* MySQL_ResultSet::init_with_stmt() doesn't need anymore the pointer to the PS
Furthermore, it isn't anymore an "init" but an end (we will rename it)
* Added MySQL_ResultSet::add_row(MYSQL_ROWS *rows) for PS only
* MySQL_Protocol::generate_pkt_row3() accepts an optional row length.
If passed, it will know the length of the row without computing it
* MyRS initialized during ASYNC_STMT_EXECUTE_STORE_RESULT_START
* Implemented throttling during ASYNC_STMT_EXECUTE_STORE_RESULT_START
* MySQL_Connection::process_rows_in_ASYNC_STMT_EXECUTE_STORE_RESULT_CONT() always
generates a heartbeat. For now unnecessary, and we will better tune it later
* test_ps_async-t.cpp tunes mysql-threshold_resultset_size to trigger buffering
More testing is required
This new function is meant to split MySQL_ResultSet::init() to later
remove all code specific to prepared statements because it assumes
it is pre-buffered
Few special queries are handled separately in ProxySQL
1. select @@version_comment limit 1
2. SHOW WARNINGS
3. SELECT USER()
4. PROXYSQL INTERNAL SESSION
5. SELECT LAST_INSERT_ID()
6. SELECT CONNECTION_ID()
7. C code mysql_set_server_option(mysql, MYSQL_OPTION_MULTI_STATEMENTS_ON)
We now remove EOF1 for these and respond with OK packets intead of EOF2
COM_STMT_PREPARE_OK contains the following packetrs
1. An OK packet with header = 0x00
2. Param Definition Block (Optional)
+ EOF Packet
3. Column Definition Block (Optional)
+ EOF Packet
Both EOF Packet are no longer sent if CLIENT_DEPRECATE_EOF
flag is enabled by client.
Signed-off-by: Bibek Shrestha <bibekshrestha@gmail.com>
When CLIENT_DEPRECATE_EOF flag is supported by client, the first EOF
packet after Metadata is not sent anymore. The EOF packet after each
ResultSet is replaced by an OK packet.
Signed-off-by: Bibek Shrestha <bibekshrestha@gmail.com>
Note that "\x8f\x80\x15" is in reverse byte order.
So the flag becomes 15808f.
15 - Authenticatin plugin length
80 - 10000000 (unused)
8f - 10001111 (the leftmost 1 is for CLIENT_SESSION_TRACK)
This is changed to 15818f.
15 - Remains the same
81 - 10000001 (the last byte says CLIENT_DEPRECATE_EOF enabled)
8f - 10001111 (remains the same)
Signed-off-by: Bibek Shrestha <bibekshrestha@gmail.com>
* Added forgotten metric increase in case of 'sync conflict' on proxysql_servers
* Added support for 'global_variables' sync in proxysql_cluster
- Added several for getting the peers to be sync.
- Added function to pull the target 'global_variables' to sync.
- Added metrics for 'global_variables' sync:
+ Number of syncs done / failed.
+ Sync conflicts.
+ Sync warnings.
* Removed unnecesary escaping for '%' char
* Fixed issue due to 'flush_mysql_variables___runtime_to_database' being called without proper locking
* Added simple helper function to get the output from a command execution
* Multiple changes to 'cluster_synct' test
+ Spawned proxysql instance now runs inside a docker container.
+ Simplified launching and logging of spawned proxysql instance.
+ Added more verbose logging about the tables state.
+ Added new tests for 'global_variables' synchronization.
* Commented several variables known to not-sync and fixed port for replica 'proxysql_server'
* Removed deprecated and non-syncable variables from 'test_cluster_sync-t'
* Added extra logging for variables sync for 'test_cluster_sync-t'
This reverts commit ee6df8ffb2.
This commit is no longer required due to a more generic impl
introduced by: '5637b31a6bd40987a383191b08d1a849df237a53'
in branch: v2.1.0-3088
- Labelling has been simplified to group metrics which
are part of a common group.
- Help has been fixed to reflect this labelling change.
- Doxygen doc and comment marks has been added to signal
metrics grouping.
This commit also add support for including more gauge metrics based on
'MySQL_Thread_status_variables', using the same approach previously used
for the 'counter' metrics.
- Metrics now follow prometheus base unit convention.
- Many metrics now make use of labels instead of enconding
the info in the metric name.
- Almost all metrics now have a meaningful 'help' message.
Former max_transaction_time is now renamed to max_transaction_idle_time.
mysql-max_transaction_idle_time defines the maximum time a transaction can stay idle.
mysql-max_transaction_time defines the maximum time of a transaction since its start.
In handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_qpo()
Changing the default of exit_after_SetParse from false to true.
If a variable isn't parsed or it needs to be forwarded, exit_after_SetParse is set to false
Renamed variable mysql-monitor_replication_lag_retries to mysql-monitor_replication_lag_count to match other variables naming convention.
As it is a counter, it minimum value is changed to 1.
Maximum value sets to 10.
Use of thread storage variable mysql_thread___monitor_replication_lag_count instead of GloMTH->variables.monitor_replication_lag_retries to avoid dirty reads .
`MySQL_Threads_Handler::listener_del` is not aware of `perthrsocks` array
used in `MySQL_Threads_Handler::listener_add`, because of this, isn't
able to properly track which descriptors are being used in each thread.
This patch makes the threads themselves responsible for closing the
descriptors that they are using for listeners.
Issue #531 added "a desperate attempt to bring back online any shunned server" when no servers are available; however, no indication or notification of this is logged. Add a log message indicating that the hostgroup has no servers available and shunned hosts are being checked before the normal shun recovery time.
Fix a mem leak in flush_ldap_variables___database_to_runtime fun of ProxySQL_Admin class when call admindb->execute_statement function.
Because in the called function(admindb->execute_statement), the strdup--strdup(sqlite3_errmsg(db)) function is used to allocate memory to store the error message of the sqlite database operation failure.
In 2.1 we are mapping the command PROXYSQL SHUTDOWN to PROXYSQL KILL = a very fast shutdown.
The legacy old PROXYSQL SHUTDOWN is now implemented as PROXYSQL SHUTDOWN SLOW
- Added new logic for making both commands actually wait
for the completion of the operations they started.
- Now "PROXYSQL STOP" command sends and OK packet to the
client on completion.
Changing the mapping logic in SESSION_TRACK_GTIDS in frontend and backend connections.
Up to now, frontend and backend `SESSION_TRACK_GTIDSs match.
This is now changed:
* backend connections are by default set to `mysql-default_session_track_gtids`
* if `mysql-default_session_track_gtids=OFF` (the default) , `session_track_gtids` is not changed on backend
* if the client asks for `session_track_gtids=OFF` , proxysql ignores it (it just acknowledge it)
* if the client asks for `session_track_gtids=OWN_GTID` , proxysql will apply it
* if the client asks for `session_track_gtids=ALL_GTIDS` , proxysql will switch to OWN_GTID and generate a warning
* if the backend doesn't support `session_track_gtids` (for example in MySQL 5.5 and MySQL 5.6), proxysql won't apply it. It knows checking server capabilities
This commit also deprecates function `MySQL_Session::handler_again___verify_backend__generic_variable()`
ProxySQL is now also able to know when proxysql binlog reader is perhaps in use tracking all gtid_port in mysql_servers.
This feature in future will be used to deprecate `mysql-default_session_track_gtids`
If isn't possible create a MySQL connection in the galera monitor
the code jump to __exit_monitor_galera_thread label, bypassing the
interr initialization.
If mysql-default_charset is changed but mysql-default_collation_connection it is not,
use mysql-default_charset to tune mysql-default_collation_connection.
Extended `mysql_variable_st` to add a new variable `is_global_variable`.
Using this variable we define the global variables.
Modified `mysql_tracked_variables[]` to also specify `is_global_variable`.
Removed `default_charset` from `MySQL_Threads_Handler::variables` to use `mysql_tracked_variables[]` instead.
Removed `default_charset` also from `mysql_thread_variables_names[]`.
Fixed `MySQL_Threads_Handler::get_variable_string()` : it was *not* returning `mysql-default_` variables.
`MySQL_Threads_Handler::set_variable()` adds `mysql-default_` variables only if `is_global_variable` is set.
In `MySQL_Threads_Handler::set_variable()` replaced a `strcasecmp()` with `strcmp()` for `mysql-default_` variables.
`MySQL_Threads_Handler::get_variables_list()` returns only `mysql-default_` variables with `is_global_variable` enabled.
`MySQL_Threads_Handler::has_variable()` for `mysql-default_` variables returns true only if variable has `is_global_variable` enabled.
`MySQL_Thread::refresh_variables()` does not refresh `mysql-default_` variables if `is_global_variable` is not enabled.
`MySQL_Thread::MySQL_Thread()` initializes to NULL all `mysql_thread___default_variables[]` variables.
Completely disabled in `ProxySQL_Admin::flush_mysql_variables___database_to_runtime()` the logic applied to all charset variables.
In the same methos was implemented the logic described in #1785 :
`mysql-default_charset` and `mysql-default_collation_connection` must be compatible:
* if only one is present, the other is configured automatically
* if not compatible, `mysql-default_charset` is configured from `mysql-default_collation_connection`
- Added extra doc about the 'status_flags' and 'server_status'.
- Added 'sql_log_bin0' and 'prepared_statement' to the 'conn.status'
from PROXYSQL SESSION INFO.