Fix a mem leak in flush_ldap_variables___database_to_runtime fun of ProxySQL_Admin class when call admindb->execute_statement function.
Because in the called function(admindb->execute_statement), the strdup--strdup(sqlite3_errmsg(db)) function is used to allocate memory to store the error message of the sqlite database operation failure.
In 2.1 we are mapping the command PROXYSQL SHUTDOWN to PROXYSQL KILL = a very fast shutdown.
The legacy old PROXYSQL SHUTDOWN is now implemented as PROXYSQL SHUTDOWN SLOW
- Added new logic for making both commands actually wait
for the completion of the operations they started.
- Now "PROXYSQL STOP" command sends and OK packet to the
client on completion.
If mysql-default_charset is changed but mysql-default_collation_connection it is not,
use mysql-default_charset to tune mysql-default_collation_connection.
Extended `mysql_variable_st` to add a new variable `is_global_variable`.
Using this variable we define the global variables.
Modified `mysql_tracked_variables[]` to also specify `is_global_variable`.
Removed `default_charset` from `MySQL_Threads_Handler::variables` to use `mysql_tracked_variables[]` instead.
Removed `default_charset` also from `mysql_thread_variables_names[]`.
Fixed `MySQL_Threads_Handler::get_variable_string()` : it was *not* returning `mysql-default_` variables.
`MySQL_Threads_Handler::set_variable()` adds `mysql-default_` variables only if `is_global_variable` is set.
In `MySQL_Threads_Handler::set_variable()` replaced a `strcasecmp()` with `strcmp()` for `mysql-default_` variables.
`MySQL_Threads_Handler::get_variables_list()` returns only `mysql-default_` variables with `is_global_variable` enabled.
`MySQL_Threads_Handler::has_variable()` for `mysql-default_` variables returns true only if variable has `is_global_variable` enabled.
`MySQL_Thread::refresh_variables()` does not refresh `mysql-default_` variables if `is_global_variable` is not enabled.
`MySQL_Thread::MySQL_Thread()` initializes to NULL all `mysql_thread___default_variables[]` variables.
Completely disabled in `ProxySQL_Admin::flush_mysql_variables___database_to_runtime()` the logic applied to all charset variables.
In the same methos was implemented the logic described in #1785 :
`mysql-default_charset` and `mysql-default_collation_connection` must be compatible:
* if only one is present, the other is configured automatically
* if not compatible, `mysql-default_charset` is configured from `mysql-default_collation_connection`
This commit prepares proxysql for further development related to SQLite3.
Specifically, it allows to either use built-in SQLite3 library, or to load it from a plugin.
Added admin variables:
* cluster_mysql_variables_save_to_disk
* cluster_admin_variables_save_to_disk
* cluster_mysql_variables_diffs_before_sync
* cluster_admin_variables_diffs_before_sync
When flush_admin_variables___database_to_runtime() is executed, flush_admin_variables___runtime_to_database() is also called to update admin variable checksum.
When flush_mysql_variables___database_to_runtime() is executed, flush_mysql_variables___runtime_to_database() is also called to update mysql variable checksum.
flush_mysql_variables___runtime_to_database() was also modified to avoid locking twice.
From 1.3.0 till 2.0.x , when COM_STMT_PREPARE is executed and Query Processor is executed, metadata are stored in MySQL_STMT_Global_info().
These metadata include routing information.
Is a prepared statement is prepared on a specific hostgroup, it will always be executed on the same hostgroup.
This causes negative consequences, like the one reported in issue #2691 .
This commit:
* remove hostgroup_id from MySQL_STMT_Global_info() and other metadata
* Query Processor is called also during COM_STMT_EXECUTE
* removes hostgroup from table stats_mysql_prepared_statements_info
* Query Processor do not perform any query rewrite ruding COM_STMT_EXECUTE
- Added new helper function to simplify ''prometheus::counters' update
and removed code duplication using it.
- 'hostgroup_manager' now exposes the same method 'p_update_metrics' as
the other modules.
- GCC 4.8 and 5 have issues with uniform initialization syntax, for
this reason, all it's uses have been replaced with assignements.
- Per-module metrics initialization functions are no longer required,
they have been replaced with the new generic ones.
libsqlinjection generates a lot of false positives.
This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints
This table can list fingerprints generated by libsqlinjection:
if the fingerprint is listed in this table, proxysql will consider it as
a false positive.
This commit also enables SQLi algorithm only if the query is not already
explicitly whitelisted.
* introduced 2 new variables:
- mysql-firewall_whitelist_enabled : defines if the firewall with whitelist algorithm is enabled or not
- firewall_whitelist_errormsg : error message returned to the client if an error message is not already set
* Query_Processor returns firewall_whitelist_mode to the calling MySQL_Session, but the logging on error log currently happens within the Query_Processor itself
* if mysql-firewall_whitelist_enabled is turned on and a user it is not found in mysql_firewall_whitelist_users , the default behavior is that query is rejected
* lookup on mysql_firewall_whitelist_users happens for username@ip (specific IP) and username@'' (all IPs) . Match on network subnet is currently not implemented
* implemented SAVE MYSQL FIREWALL FROM RUNTIME
* fixed implementation of SAVE MYSQL FIREWALL TO DISK
* runtime configuration can be viewed in runtime_mysql_firewall_whitelist_rules and runtime_mysql_firewall_whitelist_users
* upgraded libmicrohttpd to version 0.9.68
* enabled SSL in curl
* enabled SSL in libmicrohttpd compiling against GnuTLS
* SSL is now enabled in Web UI
* version checker in Web UI:
- is now able to check latest version using HTTPS
- uses the correct agent version
Implementation of new table stats_history.history_mysql_query_digest #2368
Implemented command `SAVE MYSQL DIGEST TO DISK`.
Save to disk with really minimal memory footprint compared to querying `stats_mysql_query_digest` or `stats_mysql_query_digest_reset` .
Implemented new admin variable `admin-stats_mysql_query_digest_to_disk` to automatically save to disk.
* Implementation of unit tests related to mysql_query_rules_fast_routing
* Implementation of rules_fast_routing with khash
* Adding memory usage for new fast routing algo
Extended SQLite3_result to support a mutex for concurrent rows insert.
Extensively rewritten get_query_digests() and get_query_digests_reset() to support multiple threads, and concurrently generate a resultset.
Function get_query_digests_reset() defer the deletion of objects from query digest map after generating the resultset for Admin. This drastically reduce the amount of time that query digest map is locked.
Rewritten get_query_digests_total_size() to make it multi-threaded.
Added 3 new functions for purging:
- purge_query_digests() : wrapper
- purge_query_digests_sync() : synchronous purge of query digest map, either single-threaded or multi-threaded
- purge_query_digests_async() : asynchronous purge of query digest map, single-threaded only
If table `stats_mysql_query_digest_reset` is queried but not `stats_mysql_query_digest`, only the first one is populated.
If table `stats_mysql_query_digest_reset` is queried and also `stats_mysql_query_digest`, the second one is populated as a copy from the first.
Implemented new SQL `TRUNCATE [TABLE]` for `[stats.]stats_mysql_query_digest[_reset]`: this will purge the query digest map as fast as possible without creating any table in SQLite.
For testing/benchmark:
- modified ProxySQL_Test___GenerateRandomQueryInDigestTable() to generate more random data and randomly switch `mysql_thread___query_digests_normalize_digest_text`
Extended command `PROXYSQLTEST` with the following tests:
- 1 : generates entries in digest map
- 2 : gets all the entries from the digest map, but without writing to DB, with multiple threads
- 3 : gets all the entries from the digest map and reset, but without writing to DB, with multiple threads
- 4 : purges the digest map, synchronously, in single thread
- 5 : purges the digest map, synchronously, in multiple threads
- 6 : purges the digest map, asynchronously, in single thread (this seems the faster)
Replaced some sprintf() with new function my_itoa() .
Modified class QP_query_digest_stats() to include username, schemaname and client_address if no longer than 24 bytes.
This makes storage requirement for QP_query_digest_stats() bigger, but reduces a lot of calls to memory management.
Modified QP_query_digest_stats::get_row() to drastically reduce the number of malloc/free and string copy.
Overall, this commit has the following effects:
- it increases the memory usage for query digest map. On test workload, this results in 10% more memory usage
- drastically reduces the calls to memory management
- populating `stats_mysql_query_digest_reset` is 3 times faster
- the amount of time in which query digest map is locked is reduced by 95%
Added 2 new status variables:
- queries_with_max_lag_ms__delayed
- queries_with_max_lag_ms__total_wait_time_us
Do not get replication lag from replicas if the value is 0
Fixed an error in the computation of max_lag_ms
Enhancements:
- added metrics rows_affected and rows_sent
- added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0
- added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871
Changing value at runtime causes the current file to be closed and a new one created
- fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged
Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent
Extended `eventslog_reader_sample.cpp` to support the new enhancements
After execute "PROXYSQL STOP", main() -> ProxySQL_Main_init_phase4___shutdown()
is called and GloQC is deleted, then main() -> ProxySQL_Main_init_phase2___not_started()
is called, this function starts admin module and waits on GloVars.global.start_mutex.
Before "PROXYSQL START" is executed, admin module will crash on query to stats_mysql_global.
Fixed https://github.com/sysown/proxysql/issues/2065.
- generate_mysql_servers_table() is now less verbose
- update_galera_set_writer() is protected by a mutex
- update_galera_set_writer() set status to online
- fixed wording in monitor_galera_thread()
- corrected the computing of consecutive timeout in monitor_galera_thread()
Added new command `PROXYSQL INTERNAL SESSION` that clients can execute to
receive internal information about their own connection in JSON format.
Added JSON library.
Recompiled SQLite3 to support JSON.
Added new column `extended_info` in `stats_mysql_processlist`.
Added new mysql variable `mysql-show_processlist_extended` that determine the
content of `stats_mysql_processlist.extended_info`:
- 0 : no info
- 1 : JSON format
- 2 : JSON format with pretty printing
This functionality is enabled for:
- mysql_galera_hostgroups
- mysql_group_replication_hostgroups
If writer_is_also_reader=2 and there are servers in
backup_writer_hostgroup, only these servers will be used
in reader_hostgroup
Functions add() and lookup() in MySQL_LDAP_Authentication have support for backend_username.
Added mysql_ldap_mapping table.
Created Admin::init_ldap() to be called after LDAP initialization.
Added better LDAP caching.
LOAD LDAP MAPPING TO RUNTIME cleans part of the cache (association to backend user).
All queries will have a comment "proxysql-ldap-user=%s" to track original user
Added online migration from previous table definition (1.3.0 and 1.4.0).
LOAD MYSQL USERS TO RUNTIME and SAVE MYSQL USERS TO MEMORY handle the new field
comment can be read also from config file
ProxySQL Cluster reads comment from remote node
The same applies also for:
- runtime_mysql_replication_hostgroups
- runtime_mysql_group_replication_hostgroups
- runtime_mysql_galera_hostgroups
Close#1435 and #1436
Input validation:
- client_addr not longer than INET6_ADDRSTRLEN
- % allowed only at the end of client_addr
Query rule itself remembers if there is a wildcard or not.
If there is a wildcard compares string till the wildcard.
If client_addr=='%' , it is a match all.
Statistics collected aboout GTID queries
Statistics displayed on HTTP server
Count number of GTID events per server
Online upgrade of all mysql_connections tables
Fixed path for libev
Statistics collected aboout GTID queries
Statistics displayed on HTTP server
Count number of GTID events per server
Online upgrade of all mysql_connections tables
Fixed path for libev
This is the first commit to pull data from proxysql_mysqlbinlog.
It is still in Alpha phase, as it is missing a log of important logics, like error handling, retry mechanism, timeouts, etc
* change datatype mysql_replication_hostgroups.comment to be VARCHAR NOT NULL DEFAULT '' instead of VARCHAR
* handle cases in which mysql_replication_hostgroups.comment is NULL
Handling of prepared statements changed a lot in 1.4 , as a lot of code was rewritten.
Old code was still present, and was possible to toggle it on and off based on PROXYSQL_STMT_V14 .
Because only the new code is maintained, all references to old code are now removed,
including PROXYSQL_STMT_V14
Handling of prepared statements changed a lot in 1.4 , as a lot of code was rewritten.
Old code was still present, and was possible to toggle it on and off based on PROXYSQL_STMT_V14 .
Because only the new code is maintained, all references to old code are now removed,
including PROXYSQL_STMT_V14
This is a global variable tha can be defined only in config file
When proxysql dies for whatever reason, the script defined in execute_on_exit_failure is executed.
Use this script to generate an alert.
Explicitly disabled https from libmicrohttpd (for now)
Created key and certificate, although not in use (for now)
Embedded font-awesome CSS
Added support for time range for MySQL and System
Added support for *_hour tables
Enabled digest auth in web ui (for now hardcoded credential, will fix in future commits)
Drafted a home page (not ready yet)
* all commands expected in Admin are serialized
* Cluster get mysql servers information directly from MyHGM
* Before applying MySQL servers changes, Cluster fetch again the checksum to verify it didn't change
Added variable for SQLite3 Server
Added new command:
* LOAD SQLITESERVER VARIABLES FROM MEMORY / TO RUNTIME
* LOAD SQLITESERVER VARIABLES FROM DISK / TO MEMORY
* SAVE SQLITESERVER VARIABLES FROM RUNTIME / TO MEMORY
* SAVE SQLITESERVER VARIABLES FROM MEMORY / TO DISK
Connections to SQLite3 Server uses the same MySQL users in `mysql_users`
Fixed minor issues related to ClickHouse Serve
Fixed also some Makefile errors
Set dynmic hostname/port
New commands:
* LOAD CLICKHOUSE VARIABLES FROM MEMORY / TO RUNTIME
* LOAD CLICKHOUSE VARIABLES FROM DISK / TO MEMORY
* SAVE CLICKHOUSE VARIABLES FROM RUNTIME / TO MEMORY
* SAVE CLICKHOUSE VARIABLES FROM MEMORY / TO DISK
In `MySQL_Cluster` class added functions to sync with remote node, like:
- `pull_mysql_query_rules_from_peer()`
- `pull_mysql_servers_from_peer()`
- `pull_mysql_users_from_peer()`
- `pull_proxysql_servers_from_peer()`
Added 8 new global variables in Admin.
4 variables determine after how many different checks the remote configuration will be synced:
- cluster_mysql_query_rules_diffs_before_sync
- cluster_mysql_servers_diffs_before_sync
- cluster_mysql_users_diffs_before_sync
- cluster_proxysql_servers_diffs_before_sync
4 variables determine if after a remote sync the changes need to be written to disk:
- cluster_mysql_query_rules_save_to_disk
- cluster_mysql_servers_save_to_disk
- cluster_mysql_users_save_to_disk
- cluster_proxysql_servers_save_to_disk
Table `proxysql_servers` is now automatically loaded from disk to memory and into runtime at startup.
Added new Admin's command `LOAD PROXYSQL SERVERS FROM CONFIG` to load `proxysql_servers` from config file to memory (not runtime).
Internal structures with credentials in MySQL_Authentication moved from `unsorted_map` to `map` : this to ensure the right order when generating the checksum.
Config file supports both `address` and `hostname` for `mysql_servers` and `proxysql_servers` , #1091
`ProxySQL_Admin::load_proxysql_servers_to_runtime()` now has a lock or no lock option, to avoid deadlock
For now, Cluster module is quite verbose.
Extended class ProxySQL_Checksum_Value() in ProxySQL_Cluster module to support further metrics
Implemeted `SELECT GLOBAL_CHECKSUM()` and relative tracking of global checksums
Added variable `admin-cluster_check_status_frequency` to check peer's global status at regular intervals
This commit introduces:
2 new tables:
* `runtime_checksums_values` : stores checksums of configurations in runtime. For now for `mysql_query_rules`, `mysql_servers` and `mysql_users`
* `stats_proxysql_servers_checksums` : when clustering is enabled, it collects all metrics from `runtime_checksums_values` from all its peers
3 new global variables that defines it checksum needs to be generated during `LOAD ... TO RUNTIME`
* `admin-checksum_mysql_query_rules`
* `admin-checksum_mysql_servers`
* `admin-checksum_mysql_users`
ProxySQL Cluster connections now have timeouts:
* 1 second timeout for CONNECT and WRITE
* 60 seconds timeout for READ (useful for long poll)
Extending mysql_query_rules with a new field: OK_msg
If OK_msg is not NULL, an OK packet is sent to the client with an optional
message as specified in OK_msg itself.
If OK_msg is an not NULL empty string, an OK packet is sent to the client
without any message.
If both error_msg and OK_msg are present, error_msg is returned.
This commit includes also few minor bugs, mostly related to typo that would
prevent online upgrade of mysql_query_rules from early release of 1.4.0 .
mysql_query_rules.multiplex=2 has a special meaning: do not disable multiplex for variables in THIS QUERY ONLY
Also fixed the upgrade of mysql_query_rules from 1.3
New command `PROXYSQL FLUSH CONFIGDB` allows to close and reopen configdb (`proxysql.db` on disk).
Note:
It is important that the (new) configdb is a valid one.
A misconfigured configdb will cause undefined behaviors.
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
This commit is able to speed up LOAD MYSQL SERVERS TO RUNTIME up to 10x for very large mysql_servers table with tens of thousands of servers.
Main improvements:
* removed 1 unnecessary call to `generate_mysql_servers_table()`
* replaced 1 call to `generate_mysql_servers_table()` with UPDATE statements when necessary
* implemented new function `servers_add()` that replaces `server_add()` to perform bulk operation
* replaced calls to `sqlite3_exec()` with proper parameters bindings
* use of multirows INSERT statements (32 rows at the time)
Miror improvements:
* added new variable mysql-hostgroup_manager_verbose , 1 by default for backward compatibility and debugging
* in not debug mode, access to HGM db has no shared cache
* in not debug mode, tables in HGM have checks disabled
Further note:
* Once issue #797 is completed, `mysql_servers` in MyHGM can be simplified further
New variable admin-hash_passwords , default true
When set, passwords in mysql_users are automatically hashed in runtime
when running LOAD MYSQL USERS TO RUNTIME.
For backward compatibility, passwords in mysql_users in memory and disk aren't
automatically hashed. Although, they can be easily hashed running
SAVE MYSQL USERS TO MEMORY (and eventually SAVE MYSQL USERS TO DISK)
immediately after running LOAD MYSQL USERS TO RUNTIME
Command "LOAD SCHEDULER FROM CONFIG" is now implemented.
Config file can also be read with --initial, --reload, and when a db file is not present yet.
An example was added in src/proxysql.cfg
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
The check if the configdb was not existing wasn't present anymore.
This probably happened when introducing the new LOAD FROM CONFIG commands.
Now the functionality is back.
Now Admin supports the configuration of mysql_replication_hostgroups from configfile.
That means that also LOAD MYSQL SERVERS FROM CONFIG is responsible for configuring mysql_replication_hostgroups.
This commit also allows comments in config file for `mysql_servers` and `mysql_replication_hostgroups` , related to #643
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
Scheduler is being implemented as part of the Admin module.
Added table scheduler
Added table runtime_scheduler
Implemented commands:
- [LAD|SAVE] SCHEDULER [FROM|TO] [DISK|MEMORY|RUNTIME]
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
These two new columns will be used to:
- enable SSL on a per server basis
- define maximum latency for a server to be consider usable: this will be a key element for multi DC support
mysql_query_rules table is changing from version to version.
In order to provide online upgrade, a new function was added:
- ProxySQL_Admin::disk_upgrade_mysql_query_rules()
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules