* Introduced 'ps_type' enum to differentiate between preparing a statement and executing a statement
* Fixed issue where the warning count fails to reset when the SET statement generates a warning.
* Resolved the issue where the 'server_status' and 'warning count' were not accurately represented in the statement.
* Replaced 'mysql_warning_count' with 'mysql_stmt_warning_count' for retrieving the warning count in statements.
* Added 'handle_warning' field in hostgroup_attributes TAP test
* Updated warnings TAP test
* 'MySQL_Connection::warning_count' will be globally employed for managing multiplexing control and transmitting the warning count to the client
* Fixed prepare statement bug: When preparing, if the digest matches, the result set is retrieved from the cache and sent to the client.
Note:
Currently using 'mysql_warning_count' to retrieve the warning count for prepared statement.
* Introduced global variable 'mysql-handle_warnings' to control warning support in ProxySQL
* Included 'handling_warnings' flag in the 'mysql_hostgroup_attributes' table (hostgroup_settings). This flag holds top priority and can override the value of the global variable 'mysql_handle_warnings'.
* Added two new variables to the 'stats_memory_metrics' table:
prepare_statement_metadata_memory
prepare_statement_backend_memory
* Introduced corresponding variables in the Prometheus exporter to measure memory usage of prepare statement metadata and prepare statement backend.
* Implemented 'SHOW WARNINGS' and 'SHOW COUNT(*) FROM WARNINGS' query handling.
* When a query produces warnings, the warning count is now included in the query response.
Additionally, a new variable 'mysql-query_cache_with_warnings_support' has been introduced for the query cache feature:
* A value of '0' means that if a query execution produces warnings, the resultset will not be saved in the query cache.
* A value of '1' indicates that if a query produces warnings, the resultset will be saved in the query cache, but the warning count in that resultset will be set to '0'.
** Note:
* Warning handling is dependent on the query digest and 'mysql-query_digests' must be enabled for it to work.
* Warnings saved in logfile ['mysql-log_mysql_warnings_enabled'] and the current feature can work in conjunction with each other.
MySQL client check command for dollars quote support ('SELECT $$'), is
now intercepted in both MySQL and Admin interfaces. The error reported
to the client is dependent of 'mysql-server_version' variable value.
These commands are now forwarded to the backend connections from a
session with potential transaction statuses, using the following
priority order:
1. Connections flagged with 'SERVER_STATUS_IN_TRANS', or
'autocommit=0' when 'autocommit_false_is_transaction' is set.
2. Connections with 'autocommit=0' holding a 'SAVEPOINT'.
3. Connections with 'unknown transaction status', e.g: connections
with errors.
- Improved readability for the locked map acquisition for function
'search_rules_fast_routing_dest_hg'.
- Improved function documentation and added note about usage.
- Simplified checksum computation for monitoring actions
- Refactored checksum computation for 'mysql_servers' and
'mysql_servers_v2' modules in 'commit'.
Config should only be used for checksum computation and cluster
resultset generation for 'LOAD MYSQL SERVERS TO RUNTIME'. Otherwise
user config ('main.mysql_servers') would be promoted at each 'commit'
call, and propagated through the cluster members.
- Resultsets for 'runtime_mysql_servers' and 'mysql_servers_v2' are now
directly generated by the SQL query.
- Checksum computations are simplified to 'SQLite3_result::raw_checksum'
and equivalent function 'mysql_raw_checksum'.
- Preserve old simulator behavior via 'TEST_AURORA_RANDOM' macro.
- Add new deterministic behavior for testing particular scenarios.
- Add support for simulating multiple clusters via an updated
'REPLICA_HOST_STATUS' table and hostgroup based selection.
- Provide a sensible default population for manual testing.
- Refactor of 'commit' checksums computation.
- Refactor update of 'AWS_Aurora_Hosts_resultset'.
- Fix invalid use of SHUNNED|SHUNNED_REPLICATION_LAG servers for
checksum computation via 'MYSQL_SERVERS_CHECKSUM_QUERY'.
Variable mysql-set_parser_algorithm defines which algorithm is used by SetParser
for the parsing of SET statements.
If value is 1: parse1() is used
If value is 2: parse1v2() is used
parse1v2() allows also the using a single SetParser per MySQL_Thread to
process all SET statements. Regex is compiled only once.
set_testing-240-t was also modified to test both algorithms
log_queries_not_using_indexes is not enabled because in MySQL it is *only*
a global variable, while in MariaDB is a global *and* session variable .
We believe it is not the time to create a lot of exceptions and complex logic
for conflicting backend implementations
* 'admin-ssl_keylog_file' variable has been introduced to allow users to specify the path for the SSLKEYLOG file. The file path can be an absolute path or relative to the ProxySQL data directory.
* Assigning an empty path to the 'admin-ssl_keylog_file' variable signifies that the SSLKEYLOG file feature is disabled.
* In case an invalid path is provided for the SSLKEYLOG file, ProxySQL will automatically revert to the last valid path value.
* 'PROXYSQL FLUSH LOGS' command can be used to rotate SSLKEYLOG file.
# Conflicts:
# lib/mysql_connection.cpp
Session 'status' and 'client_myds->DSS' should never be modified for a
regular session prior to calling 'RequestEnd'. It's 'RequestEnd'
responsibility to appropriately change these statuses after the rest of
cleanup operations for the request has taken place, including logging.
To enable coredump feature, specify the filename and line number in the coredump_filters table.
Example:
INSERT INTO coredump_filters (filename,line) VALUES ('mysql_data_stream.cpp', 498);
LOAD COREDUMP TO RUNTIME;
Note:
* It is important to note that coredump filters cannot be persisted on disk and will always be empty when a new instance is created.
* Filenames are case-sensitive
* 'admin-coredump_generation_threshold', has been introduced to specify the maximum number of core files that can be generated.
* 'admin-coredump_generation_interval_ms', has been implemented to define the time gap between the generation of core files.
Extended account_details_t to also store clear_text_password if client
provides it during a full authentication in caching_sha2_password .
De facto, this implements the "caching" part of caching_sha2_password .
Now mysql_users.password can store the authentication_string as
in MySQL's mysql.user table.
Note: this commit also changes SALT_LEN_MAX in SHA-crypt code from
16 to 20 because MySQL caching_sha2_password uses 20 bytes salt
In case an autodiscovered server is found 'OFFLINE_HARD', the recovery
logic now properly updates the 'servers_defaults' values for the server,
in the same way as for a new discovered server.
- Add support for autodiscovery for GR for MySQL8.
- Allow to configure autodiscovered servers defaults via
'mysql_hostgroup_attributes'.
- Add support for GR autodiscovery for Cluster Simulator.
- Improve server 'viable_candidate' detection for MySQL 8,
'RECOVERING' state is now also considered.
- Add parameter for conditionally locking inside function.
- Move potential syscalls outside the 'pthread_rwlock_rdlock' section.
- Reduce number of 'sprintf' calls.
- Improved PROXYSQLTEST 11/15 with data randomly exceeding function
default buffer size.
This is an alternative implementation of commit cfb34ef256
because it was failing to compile on clang 14 , that was complaining about
a missing unnamed destructor
- Thread variable has been replaced for new variable in 'Query_Processor'.
- Computation for 'mysql_query_rules_memory' is now performed by Admin
during 'load_mysql_query_rules_to_runtime' instead of lazily by the
MySQL_Threads.
- Implemented new variable 'query_rules_fast_routing_algorithm' that
allows choosing between the classic per-thread hashmaps for
'query_rules_fast_routing' searches (1), and a new unique global
hashmap (2).
- Improved locking time on 'QueryProcessor::rwlock' during
'load_mysql_query_rules_to_runtime'.
- Changed computation of 'mysql_query_rules_memory' to be lazily
updated by threads themselves while building the per-threads maps.
- Fixed 'ProxySQL_Admin::stats___memory_metrics' report of the memory
metrics.
- Improved 'PROXYSQLTEST 17' command, for being able to micro-benchmark
and check 'query_rules_fast_routing_algorithm'.
* Added separate fetching of mysql_server_incoming and runtime_mysql_server records based on algorithm selection.
* Few memory leaks fix
* Code refactoring
Introduced new schema debugdb_disk (on disk) and table debug_log .
Debug entries will now go to this new table based on the value of
variable admin-debug_output . If:
1) default behavior: write to stderr
2) write only to debugdb_disk.debug_log
3) 3 = 1+2 = write to stderr and to debugdb_disk.debug_log
All these functionalities exist only on debug build.
Table debugdb_disk.debug_log has also a column named "note" that can be used
to write user notes during debugging.
Otherwise, all queries processed by ProxySQL have to wait for
get_query_digest() to finish reading the maps.
With this patch, ProxySQL stores query digests in the auxiliary maps
while the main maps are being used by get_query_digests(). Once
get_query_digests() finish reading the main maps, it merge the auxiliary
maps in to the main maps and clear the content of the auxiliary maps.
This commit optimizes the read-only actions method by removing the dependency on proxysql admin tables. Instead, a separate mapping container is maintained, which records the state of the HGM servers. This avoids the need to call the commit method on every change, which improves performance.
The changes in this commit also ensure that the mapping container is properly updated when servers are added, removed or modified.
When receiving MYSQL_COM_BINLOG_DUMP , MYSQL_COM_BINLOG_DUMP_GTID or
MYSQL_COM_REGISTER_SLAVE , proxysql automatically switches to fast_forward .
Although it is important to correctly configure "encrypted" in the backend
data stream if needed.
Used PMC-10005 for reference , as this is implemented similarly in many
parts of the code.
During the switch proxy_info() now also reports the client and the
command received.
The destination hostgroup assigned from previous COM_QUERY commands is
the one used to establish the fast_forward connection. As a result,
multiple binlog clients can consume binlog events from different MySQL
hostgroups using the same username.
If mysql_hostgroup_attributes.multiplex:
- 0 : multiplexing is always disabled
- 1 : multiplexing is enabled for the hostgroup, but other criteria can disable multiplexing (transaction, temp tables, query rules, etc)
When Query Cache entries reach the soft ttl, the next query hit the
backend and refresh the entry. While the refresh happens, other queries
keeps getting the "old" Query Cache entry.
Soft ttl is a percentage of the cache ttl and is defined by a global
variable for all the entries. If the value is 0, soft ttl is disabled
and no used.
In Admin:
- replaced a lot of flush_[...]__from_memory_to_disk() and flush_[...]__from_disk_to_memory() with a generic flush_GENERIC__from_to()
- several commands like "LOAD MYSQL SERVERS FROM DISK" and similar are automatically generated for various modules (not all) and saved/map in a map load_save_disk_commands
- FlushCommandWrapper() is able to map the various LOAD/SAVE from/to DISK into commands to run on SQLite
This is a draft of a rework for Monitor 'Group Replication', this
reworks reuses the Monitoring model already taken for AWS Aurora for
'Group Replication'.
Asynchronous handling of Ping, Group replication, Replication Lag, Read Only and Galera if MySQL connection is available, else tasks will be delegated to Consumer Thread.
This commit pack a couple of fixes and improvements for the RESTAPI:
- Homogenization of GET/POST endpoint responses by ProxySQL. All
responses should now be guaranteed to be valid JSON.
- Fix JSON construction from parameters supplied to GET endpoint.
- Add two new fields 'script_stdout' and 'script_stderr' to the
JSON response when the target script fails to be executed. This is,
when it exists with an error code other than zero. This makes the
response homogeneous to when the scripts fails to produce a valid
JSON output, and add extra information for debugging on client side.
- Add a new debugging module 'PROXY_DEBUG_RESTAPI', to help tracing in
debugging builds the requests to the endpoints.
This commit addresses several issues with 'wexecvp':
- Fix invalid double call to 'close' in case fd was higher than
'FD_SETSIZE'. This could lead to the invalidation of an unrelated
fd resulting in asserts, as in issue #4001 and other instabilities.
- Fix previous limitations of the legacy 'select 'impl that rendered
the RESTAPI unusable when ProxySQL had more then 'FD_SETSIZE' fds
opened.
- Other minor improvements in function logic and interface.
* Added forced triggering of DNS Cache Update on MySQL Servers and ProxySQL Servers table update.
* Direct DNS cache update via socket (connected peer ip) if record is not available in cache.
* Used strdup instead of realloc.
* DNS lookup only once for ProxySQL Cluster Nodes.
* Added multiple IP support with load balancing (Round Robin Scheduling)
* Added DNS resolver request max queue size
* Added AI_ADDRCONFIG flag
* Exception handling
This commit also introduces the following changes:
- Fix invalid formatting of 'stats_mysql_users' when username exceeds
'256' characters.
- Improve error handling in utility function 'string_format'.
- Add new utility functions 'cstr_format', as improved versions of
previous 'string_format' function.
Removed dependency from installed curl and openssl: now it uses
statically linked ones.
Updated config.guess and config.sub for cityhash and libdaemon
MariaDB client connector uses statically linked libssl and installed
libiconv
DEBUG build uses pthread_threadid_np() instead of syscall(SYS_gettid)
Fixed some variable types.
Enabled the use of jemalloc also for MacOS
Alias mallctl() to je_mallctl()
Removed libre2.a from libproxysql.a
This commit contains an implementation rework and a fix:
- Impl for 'auto_increment_delay_multiplex_timeout_ms' has been
reworked in favor of reusing 'wait_until' to share logic with
previous 'connection_delay_multiplex_ms' variable.
- Fix previous 'connection_delay_multiplex_ms' behavior preventing
connection retaining when traffic unrelated to target hostgroup is
being received by the session.
Because of MySQL bug https://bugs.mysql.com/bug.php?id=107875 ,
autocommit=0 can "create a transaction" even if SERVER_STATUS_IN_TRANS
is not set. This means that a SAVEPOINT can be created in a "transaction"
created with autocommit=0 but without SERVER_STATUS_IN_TRANS .
This commit takes also care of scenarios when autocommit=0 was switched back
to autocommit=1 when a SELECT was executed while variable
mysql-enforce_autocommit_on_reads was false.
MySQL_Session::NumActiveTransactions() now accepts a flag to check
for savepoints.
Query 'CLUSTER_QUERY_MYSQL_SERVERS' is now responded by 'runtime_mysql_servers'
as it previously was. Documentation reasoning about the change has been added
to 'MySQL_HostGroups_Manager::runtime_mysql_servers'.
- Cluster now syncs the server tables via 'incoming_*' tables generated
during 'load_mysql_servers_to_runtime'.
- Cluster now syncs 'mysql_users' table via a resultset generated via
'__refresh_users'.
Multiple changes:
- Query_Processor stored the resultset of query rules loaded to runtime
(previously this was only for query rules fast routing)
- Admin returns the stored resultset (query rules and fast routing)
when queries by Cluster
- In khash replaced the hashing function from the built-in
__ac_X31_hash_string to CityHash32
- When Cluster is used, it calls load_mysql_query_rules_to_runtime() passing
the resultsets retrieved by the remote peer
- Increased SQLite cache_size to ~50MB: this seems to be a very small optimization
and probably it will be reverted
- pull_mysql_query_rules_from_peer() uses transactions to write
to mysql_query_rules_fast_routing
- (important change) pull_mysql_query_rules_from_peer() verifies the checksum
of MySQL Query ules before loading them to runtime
The following cluster modules now compute their expected checksums after fetch:
- mysql_query_rules
- mysql_users
- mysql_servers
- mysql_global_variables
If variable mysql-monitor_replication_lag_group_by_host=false: (default)
Monitor will perform 1 replication lag check per server per hostgroup.
If variable mysql-monitor_replication_lag_group_by_host=true:
Monitor will perform 1 replication lag check per server.
This variable need to be set only in setups in which the same server is
configured in many hostgroups, thus reducing the number of checks
This is now used in ProxySQL_Admin and MySQL_HostGroups_Manager.
It improves readibility and it makes the code less error prone.
Drawback: queries are generated at runtime.
It is possible that in future we will replace this with some strings
initialization during bootstrap.
- Add new version of macros proxy_error|proxy_warning|proxy_info.
- Add new tables 'stats_proxysql_message_metrics' and
'stats_proxysql_message_metrics_reset'.
- Add new prometheus metric family 'proxysql_message_count_total'.
- Monitoring actions 'ping' and 'connect' now make use of a new
introduced table 'monitor.mysql_servers' instead of 'mysql_servers'
table from Admin.
- This table is kept in sync via a shared resulset in
'MySQL_HostGroups_Manager'.
New variables supported:
- lc_messages
- lc_time_names
- long_query_time
- max_execution_time
- max_heap_table_size
- max_sort_length
- optimizer_prune_level
- optimizer_search_depth
- optimizer_switch
- sort_buffer_size
- sql_big_selects
- timestamp
- tmp_table_size
Note: sql_big_selects is still not fully supported when
combined with max_join_size.
In set testing TAP ,set large max_execution_time.
If max_execution_time is too small, the TAP test may fails because the
queries will timeout.
Better parsing of SET SQL_LOG_BIN .
SQL_LOG_BIN is now managed by the same functions that handle boolean variables
instead of the now deprecated MySQL_Session::handler_WCD_SS_MCQ_qpo_Parse_SQL_LOG_BIN()
For naming convention, enum value SQL_LOG_BIN was renamed to SQL_SQL_LOG_BIN
In MySQL_Thread we also deprecated match_regexes[0] , previously
used to parse `set sql_log_bin`.
Functions Variable::fill_server_internal_session() and
Variable::fill_client_internal_session() seems to handle sql_log_bin as a
special value. This exception was removed.
Fixed a bug in MySQL_Session::generate_proxysql_internal_session_json() where
when backend variables needed to be generated, fill_client_internal_session()
was incorrectly called instead of fill_server_internal_session() , leading to
incorrect values reported by `PROXYSQL INTERNAL SESSION`
- Provided a new function to partially initialize 'MySQL_ResultSet'.
- Replaced 'generate_pkt_row' in favor of 'generate_pkt_row3' for
resulset writing.
Deprecated variables:
- net_write_timeout
Renamed set_testing-t.csv to set_testing-t.csv.obsolete , and removed net_write_timeout from set_testing-t.csv
Variables added to ignore list:
- net_write_timeout
- net_buffer_length
- read_buffer_size
- read_rnd_buffer_size
Added support for quoting with backtick
Renamed `enum variable_name` into `enum mysql_variable_name` for better readibility
`enum mysql_variable_name` has variable names alphabetically ordered after SQL_NAME_LAST_LOW_WM
`mysql_tracked_variables` has variable names alphabetically ordered after SQL_NAME_LAST_LOW_WM
A lot of new variables added in `mysql_tracked_variables` , although not all tracked yet (TODO)
In `MySQL_Variables::MySQL_Variables()`:
- if `internal_variable_name` in `mysql_tracked_variables` is NULL , it will be automatically initialized
- it performs some sanity check between `enum mysql_variable_name` and `mysql_tracked_variables`
Testing:
- refactored code in set_testing-t.cpp and set_testing-multi-t.cpp
- added set_testing-240-t.cpp to test improvements in 2.4.0
- generate_set_session_csv to automatically create set_testing-240.csv used by set_testing-240-t
* Fixed typo for variable_id in history_mysql_status_variables table definition to be INT NOT NULL as intended.
* Added comments before debug ifdefs to explain their purpose
* Optimized get_variable_id_for_name by performing one less search by using iterator insteaed of knows_variable_name() call.
* Fixed memory leak by deleting resultset before return in lambda.
Changed the get_variable_id_for_name, and knows_variable_name to use const string &. Changed immutable strings to to be const as well. Added details comment to test_admin_stats-t.cpp test. In the test code, improved documentation, set a valid interval for metrics insertion, and added the commands to set the interval. Changed the sleep time to be the inteval plus a second should be long enough to let new metrics data be added. Put additional sleep before test 2. Fixed incorrect column index usage in test 4 result set. Reformulated test 5, to check the number of records inserted after new interval is correct.
Rahim Kanji
René Cannaò
Miro Stauder
Javier Jaramago Fernández
Phil Porada
Samantha
Thomas Saunders
Javier Sánchez Parra
Jared Jetsel
Miro Stauder
Miro Stauder
André Cabaça