* Added multiple IP support with load balancing (Round Robin Scheduling)
* Added DNS resolver request max queue size
* Added AI_ADDRCONFIG flag
* Exception handling
This commit also introduces the following changes:
- Fix invalid formatting of 'stats_mysql_users' when username exceeds
'256' characters.
- Improve error handling in utility function 'string_format'.
- Add new utility functions 'cstr_format', as improved versions of
previous 'string_format' function.
Removed dependency from installed curl and openssl: now it uses
statically linked ones.
Updated config.guess and config.sub for cityhash and libdaemon
MariaDB client connector uses statically linked libssl and installed
libiconv
DEBUG build uses pthread_threadid_np() instead of syscall(SYS_gettid)
Fixed some variable types.
Enabled the use of jemalloc also for MacOS
Alias mallctl() to je_mallctl()
Removed libre2.a from libproxysql.a
This commit contains an implementation rework and a fix:
- Impl for 'auto_increment_delay_multiplex_timeout_ms' has been
reworked in favor of reusing 'wait_until' to share logic with
previous 'connection_delay_multiplex_ms' variable.
- Fix previous 'connection_delay_multiplex_ms' behavior preventing
connection retaining when traffic unrelated to target hostgroup is
being received by the session.
Because of MySQL bug https://bugs.mysql.com/bug.php?id=107875 ,
autocommit=0 can "create a transaction" even if SERVER_STATUS_IN_TRANS
is not set. This means that a SAVEPOINT can be created in a "transaction"
created with autocommit=0 but without SERVER_STATUS_IN_TRANS .
This commit takes also care of scenarios when autocommit=0 was switched back
to autocommit=1 when a SELECT was executed while variable
mysql-enforce_autocommit_on_reads was false.
MySQL_Session::NumActiveTransactions() now accepts a flag to check
for savepoints.
Query 'CLUSTER_QUERY_MYSQL_SERVERS' is now responded by 'runtime_mysql_servers'
as it previously was. Documentation reasoning about the change has been added
to 'MySQL_HostGroups_Manager::runtime_mysql_servers'.
- Cluster now syncs the server tables via 'incoming_*' tables generated
during 'load_mysql_servers_to_runtime'.
- Cluster now syncs 'mysql_users' table via a resultset generated via
'__refresh_users'.
Multiple changes:
- Query_Processor stored the resultset of query rules loaded to runtime
(previously this was only for query rules fast routing)
- Admin returns the stored resultset (query rules and fast routing)
when queries by Cluster
- In khash replaced the hashing function from the built-in
__ac_X31_hash_string to CityHash32
- When Cluster is used, it calls load_mysql_query_rules_to_runtime() passing
the resultsets retrieved by the remote peer
- Increased SQLite cache_size to ~50MB: this seems to be a very small optimization
and probably it will be reverted
- pull_mysql_query_rules_from_peer() uses transactions to write
to mysql_query_rules_fast_routing
- (important change) pull_mysql_query_rules_from_peer() verifies the checksum
of MySQL Query ules before loading them to runtime
The following cluster modules now compute their expected checksums after fetch:
- mysql_query_rules
- mysql_users
- mysql_servers
- mysql_global_variables
If variable mysql-monitor_replication_lag_group_by_host=false: (default)
Monitor will perform 1 replication lag check per server per hostgroup.
If variable mysql-monitor_replication_lag_group_by_host=true:
Monitor will perform 1 replication lag check per server.
This variable need to be set only in setups in which the same server is
configured in many hostgroups, thus reducing the number of checks
This is now used in ProxySQL_Admin and MySQL_HostGroups_Manager.
It improves readibility and it makes the code less error prone.
Drawback: queries are generated at runtime.
It is possible that in future we will replace this with some strings
initialization during bootstrap.
- Add new version of macros proxy_error|proxy_warning|proxy_info.
- Add new tables 'stats_proxysql_message_metrics' and
'stats_proxysql_message_metrics_reset'.
- Add new prometheus metric family 'proxysql_message_count_total'.
- Monitoring actions 'ping' and 'connect' now make use of a new
introduced table 'monitor.mysql_servers' instead of 'mysql_servers'
table from Admin.
- This table is kept in sync via a shared resulset in
'MySQL_HostGroups_Manager'.
New variables supported:
- lc_messages
- lc_time_names
- long_query_time
- max_execution_time
- max_heap_table_size
- max_sort_length
- optimizer_prune_level
- optimizer_search_depth
- optimizer_switch
- sort_buffer_size
- sql_big_selects
- timestamp
- tmp_table_size
Note: sql_big_selects is still not fully supported when
combined with max_join_size.
In set testing TAP ,set large max_execution_time.
If max_execution_time is too small, the TAP test may fails because the
queries will timeout.
Better parsing of SET SQL_LOG_BIN .
SQL_LOG_BIN is now managed by the same functions that handle boolean variables
instead of the now deprecated MySQL_Session::handler_WCD_SS_MCQ_qpo_Parse_SQL_LOG_BIN()
For naming convention, enum value SQL_LOG_BIN was renamed to SQL_SQL_LOG_BIN
In MySQL_Thread we also deprecated match_regexes[0] , previously
used to parse `set sql_log_bin`.
Functions Variable::fill_server_internal_session() and
Variable::fill_client_internal_session() seems to handle sql_log_bin as a
special value. This exception was removed.
Fixed a bug in MySQL_Session::generate_proxysql_internal_session_json() where
when backend variables needed to be generated, fill_client_internal_session()
was incorrectly called instead of fill_server_internal_session() , leading to
incorrect values reported by `PROXYSQL INTERNAL SESSION`
- Provided a new function to partially initialize 'MySQL_ResultSet'.
- Replaced 'generate_pkt_row' in favor of 'generate_pkt_row3' for
resulset writing.
Deprecated variables:
- net_write_timeout
Renamed set_testing-t.csv to set_testing-t.csv.obsolete , and removed net_write_timeout from set_testing-t.csv
Variables added to ignore list:
- net_write_timeout
- net_buffer_length
- read_buffer_size
- read_rnd_buffer_size
Added support for quoting with backtick
Renamed `enum variable_name` into `enum mysql_variable_name` for better readibility
`enum mysql_variable_name` has variable names alphabetically ordered after SQL_NAME_LAST_LOW_WM
`mysql_tracked_variables` has variable names alphabetically ordered after SQL_NAME_LAST_LOW_WM
A lot of new variables added in `mysql_tracked_variables` , although not all tracked yet (TODO)
In `MySQL_Variables::MySQL_Variables()`:
- if `internal_variable_name` in `mysql_tracked_variables` is NULL , it will be automatically initialized
- it performs some sanity check between `enum mysql_variable_name` and `mysql_tracked_variables`
Testing:
- refactored code in set_testing-t.cpp and set_testing-multi-t.cpp
- added set_testing-240-t.cpp to test improvements in 2.4.0
- generate_set_session_csv to automatically create set_testing-240.csv used by set_testing-240-t
* Fixed typo for variable_id in history_mysql_status_variables table definition to be INT NOT NULL as intended.
* Added comments before debug ifdefs to explain their purpose
* Optimized get_variable_id_for_name by performing one less search by using iterator insteaed of knows_variable_name() call.
* Fixed memory leak by deleting resultset before return in lambda.
Changed the get_variable_id_for_name, and knows_variable_name to use const string &. Changed immutable strings to to be const as well. Added details comment to test_admin_stats-t.cpp test. In the test code, improved documentation, set a valid interval for metrics insertion, and added the commands to set the interval. Changed the sleep time to be the inteval plus a second should be long enough to let new metrics data be added. Put additional sleep before test 2. Fixed incorrect column index usage in test 4 result set. Reformulated test 5, to check the number of records inserted after new interval is correct.
MySQL_HostGroups_Manager::unshun_server_all_hostgroups() scans all hostgroups
looking for a specific server, and if shunned it will bring it back online
This commits partially revert 0dde4adb21
It also improve the TAP test reg_test_3493-USE_with_comment-t.cpp with a
variety of combinations, that helped find a bug in the tokenizer
mysql_query_digest_and_first_comment()
Query_Processor::get_current_query_rules_fast_routing_count() now allows to
quickly execute this query is ran on Admin:
SELECT COUNT(*) FROM runtime_mysql_query_rules_fast_routing
When shunning a node due to replication lag in a group replication cluster,
we first shun the node as MYSQL_SERVER_STATUS_SHUNNED , then we shun it
as MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG .
In this way we prevent (for a short time) to kill connections on that backend.
This backing off from that server can give the server enough time to sync up.
See discussion in comments in https://github.com/sysown/proxysql/pull/3533
Add initial support for FR limiting the number of connection errors that
can be initiated from a particular address before deniying future
connections from that address. Feature is controlled by new introduced
variables:
- mysql-client_host_error_counts
- mysql-client_host_cache_size
Command PROXYSQL RELOAD TLS is able to load new key/cert files and
create a new SSL context.
Loading of SSL can happens in two stages:
* during bootstrap: if it fails, proxysql dies
* running PROXYSQL RELOAD TLS: if it fails, proxysql will keep the old SSL context
This new method should successfully handle errors like missing or corrupted files.
We are also adding two buffers used to store the key/cert, to be used by the web interface.
This commit add the support of ERR packet in binary protocol resultset,
for example due to MAX_EXECUTION_TIME while running a prepared statement.
It also changes how EOF packets are added: instead of writing the EOF in the
resultset buffer, the EOF packet is written in its own packet. This allows to
easily remove it if needed.
It also fix what seems (not sure) a bug in MariaDB client library, where if a
prepared statement is interrupted it cannot be executed again
1. Introduced new global variable: 'monitor_groupreplication_max_transaction_behind_for_read_only',
that modifies the behavior of 'group_replication_lag'.
2. Improved logic making use of 'MyHGC_find' instead of directly
searching 'MyHostGroups' structure.
3. Improved 'group_replication_lag' documentation with new
implementation updates.
4. Introduced changes to 'update_group_replication_set_writer'
preserving writters placed in 'OFFLINE_SOFT' state.
Added function MySQL_Session::detected_broken_connection() that unifies the way broken connections are logged.
It also logs username, and last time used.
Added also a new macro proxy_error_inline()
This commit introduces a first simulator for read_only monitoring
ProxySQL itself will simulate hundreds of backends across hundreds of
hostgroups, all configured in hundreds of clusters, and return read_only
values for each of the simulated backend.
The read_only value can be configured in table READONLY_STATUS available in
the SQLite3Server module.
If a backend isn't configured in table READONLY_STATUS , its default read_only
value is 1.
It is possible to simulate a lot of simultaneous failover running queries like
the following:
UPDATE READONLY_STATUS SET read_only=1;
CREATE TABLE t1 AS SELECT hostname FROM READONLY_STATUS ORDER BY RANDOM() LIMIT 50;
UPDATE READONLY_STATUS SET read_only=0 WHERE hostname IN (SELECT hostname FROM t1);
DROP TABLE t1;
When a cluster node (client) connects to another cluster node, it
advertises its UUID but also exports its `admin-mysql_ifaces` .
The receiving cluster can use this information to try to guess how
to connect to the node that initialized the connection, for example
to collect metrics.
This is visible in table `stats_proxysql_servers_clients_status` .
HostGroups are stored in an array of pointers.
This commit allows the lookup using an unordered_map if the number of
Hostgroups is greater than 100.
For less number of hostgroups, a sequential scan seems more efficient.
This also introduce a test for micro benchmark.
This commit rewrites MySQL_HostGroups_Manager::read_only_action() in way to
reduce the calls to SQLite. It relies on two new std::set() :
- read_only_set1: if the server is already in a writer hostgroup
- read_only_set2: a cache to track the first time that a server has RO=0
Added also an index on MySQL_HostGroups_Manager,
ON mysql_servers (hostname,port)
ProxySQL will now automatically generate a UUID during start up.
The UUID can be either passed through the command line using -U or --uuid,
or saved on disk in a new table named global_settings.
The new table is used to distinguish it from global_variables that is replicated.
Table global_settings isn't replicated by Cluster.
If UUID is not present, it is automatically generated and saved.
`generate_pkt_field2` is a less generic version of `generate_pkt_field`
and it is meant to be used only for queries executed on backend.
The function takes a `MYSQL_FIELD` as one of its arguments, thus a lot of
lengths do not need to be computed
* MySQL_ResultSet() stores a pointer to the PS
* MySQL_ResultSet::init_with_stmt() doesn't need anymore the pointer to the PS
Furthermore, it isn't anymore an "init" but an end (we will rename it)
* Added MySQL_ResultSet::add_row(MYSQL_ROWS *rows) for PS only
* MySQL_Protocol::generate_pkt_row3() accepts an optional row length.
If passed, it will know the length of the row without computing it
* MyRS initialized during ASYNC_STMT_EXECUTE_STORE_RESULT_START
* Implemented throttling during ASYNC_STMT_EXECUTE_STORE_RESULT_START
* MySQL_Connection::process_rows_in_ASYNC_STMT_EXECUTE_STORE_RESULT_CONT() always
generates a heartbeat. For now unnecessary, and we will better tune it later
* test_ps_async-t.cpp tunes mysql-threshold_resultset_size to trigger buffering
More testing is required
This new function is meant to split MySQL_ResultSet::init() to later
remove all code specific to prepared statements because it assumes
it is pre-buffered
Few special queries are handled separately in ProxySQL
1. select @@version_comment limit 1
2. SHOW WARNINGS
3. SELECT USER()
4. PROXYSQL INTERNAL SESSION
5. SELECT LAST_INSERT_ID()
6. SELECT CONNECTION_ID()
7. C code mysql_set_server_option(mysql, MYSQL_OPTION_MULTI_STATEMENTS_ON)
We now remove EOF1 for these and respond with OK packets intead of EOF2
When CLIENT_DEPRECATE_EOF flag is supported by client, the first EOF
packet after Metadata is not sent anymore. The EOF packet after each
ResultSet is replaced by an OK packet.
Signed-off-by: Bibek Shrestha <bibekshrestha@gmail.com>
* Added forgotten metric increase in case of 'sync conflict' on proxysql_servers
* Added support for 'global_variables' sync in proxysql_cluster
- Added several for getting the peers to be sync.
- Added function to pull the target 'global_variables' to sync.
- Added metrics for 'global_variables' sync:
+ Number of syncs done / failed.
+ Sync conflicts.
+ Sync warnings.
* Removed unnecesary escaping for '%' char
* Fixed issue due to 'flush_mysql_variables___runtime_to_database' being called without proper locking
* Added simple helper function to get the output from a command execution
* Multiple changes to 'cluster_synct' test
+ Spawned proxysql instance now runs inside a docker container.
+ Simplified launching and logging of spawned proxysql instance.
+ Added more verbose logging about the tables state.
+ Added new tests for 'global_variables' synchronization.
* Commented several variables known to not-sync and fixed port for replica 'proxysql_server'
* Removed deprecated and non-syncable variables from 'test_cluster_sync-t'
* Added extra logging for variables sync for 'test_cluster_sync-t'
- Labelling has been simplified to group metrics which
are part of a common group.
- Help has been fixed to reflect this labelling change.
- Doxygen doc and comment marks has been added to signal
metrics grouping.