Statistics collected aboout GTID queries
Statistics displayed on HTTP server
Count number of GTID events per server
Online upgrade of all mysql_connections tables
Fixed path for libev
Statistics collected aboout GTID queries
Statistics displayed on HTTP server
Count number of GTID events per server
Online upgrade of all mysql_connections tables
Fixed path for libev
This is the first commit to pull data from proxysql_mysqlbinlog.
It is still in Alpha phase, as it is missing a log of important logics, like error handling, retry mechanism, timeouts, etc
* change datatype mysql_replication_hostgroups.comment to be VARCHAR NOT NULL DEFAULT '' instead of VARCHAR
* handle cases in which mysql_replication_hostgroups.comment is NULL
Handling of prepared statements changed a lot in 1.4 , as a lot of code was rewritten.
Old code was still present, and was possible to toggle it on and off based on PROXYSQL_STMT_V14 .
Because only the new code is maintained, all references to old code are now removed,
including PROXYSQL_STMT_V14
Handling of prepared statements changed a lot in 1.4 , as a lot of code was rewritten.
Old code was still present, and was possible to toggle it on and off based on PROXYSQL_STMT_V14 .
Because only the new code is maintained, all references to old code are now removed,
including PROXYSQL_STMT_V14
This is a global variable tha can be defined only in config file
When proxysql dies for whatever reason, the script defined in execute_on_exit_failure is executed.
Use this script to generate an alert.
Explicitly disabled https from libmicrohttpd (for now)
Created key and certificate, although not in use (for now)
Embedded font-awesome CSS
Added support for time range for MySQL and System
Added support for *_hour tables
Enabled digest auth in web ui (for now hardcoded credential, will fix in future commits)
Drafted a home page (not ready yet)
* all commands expected in Admin are serialized
* Cluster get mysql servers information directly from MyHGM
* Before applying MySQL servers changes, Cluster fetch again the checksum to verify it didn't change
Added variable for SQLite3 Server
Added new command:
* LOAD SQLITESERVER VARIABLES FROM MEMORY / TO RUNTIME
* LOAD SQLITESERVER VARIABLES FROM DISK / TO MEMORY
* SAVE SQLITESERVER VARIABLES FROM RUNTIME / TO MEMORY
* SAVE SQLITESERVER VARIABLES FROM MEMORY / TO DISK
Connections to SQLite3 Server uses the same MySQL users in `mysql_users`
Fixed minor issues related to ClickHouse Serve
Fixed also some Makefile errors
Set dynmic hostname/port
New commands:
* LOAD CLICKHOUSE VARIABLES FROM MEMORY / TO RUNTIME
* LOAD CLICKHOUSE VARIABLES FROM DISK / TO MEMORY
* SAVE CLICKHOUSE VARIABLES FROM RUNTIME / TO MEMORY
* SAVE CLICKHOUSE VARIABLES FROM MEMORY / TO DISK
In `MySQL_Cluster` class added functions to sync with remote node, like:
- `pull_mysql_query_rules_from_peer()`
- `pull_mysql_servers_from_peer()`
- `pull_mysql_users_from_peer()`
- `pull_proxysql_servers_from_peer()`
Added 8 new global variables in Admin.
4 variables determine after how many different checks the remote configuration will be synced:
- cluster_mysql_query_rules_diffs_before_sync
- cluster_mysql_servers_diffs_before_sync
- cluster_mysql_users_diffs_before_sync
- cluster_proxysql_servers_diffs_before_sync
4 variables determine if after a remote sync the changes need to be written to disk:
- cluster_mysql_query_rules_save_to_disk
- cluster_mysql_servers_save_to_disk
- cluster_mysql_users_save_to_disk
- cluster_proxysql_servers_save_to_disk
Table `proxysql_servers` is now automatically loaded from disk to memory and into runtime at startup.
Added new Admin's command `LOAD PROXYSQL SERVERS FROM CONFIG` to load `proxysql_servers` from config file to memory (not runtime).
Internal structures with credentials in MySQL_Authentication moved from `unsorted_map` to `map` : this to ensure the right order when generating the checksum.
Config file supports both `address` and `hostname` for `mysql_servers` and `proxysql_servers` , #1091
`ProxySQL_Admin::load_proxysql_servers_to_runtime()` now has a lock or no lock option, to avoid deadlock
For now, Cluster module is quite verbose.
Extended class ProxySQL_Checksum_Value() in ProxySQL_Cluster module to support further metrics
Implemeted `SELECT GLOBAL_CHECKSUM()` and relative tracking of global checksums
Added variable `admin-cluster_check_status_frequency` to check peer's global status at regular intervals
This commit introduces:
2 new tables:
* `runtime_checksums_values` : stores checksums of configurations in runtime. For now for `mysql_query_rules`, `mysql_servers` and `mysql_users`
* `stats_proxysql_servers_checksums` : when clustering is enabled, it collects all metrics from `runtime_checksums_values` from all its peers
3 new global variables that defines it checksum needs to be generated during `LOAD ... TO RUNTIME`
* `admin-checksum_mysql_query_rules`
* `admin-checksum_mysql_servers`
* `admin-checksum_mysql_users`
ProxySQL Cluster connections now have timeouts:
* 1 second timeout for CONNECT and WRITE
* 60 seconds timeout for READ (useful for long poll)
Extending mysql_query_rules with a new field: OK_msg
If OK_msg is not NULL, an OK packet is sent to the client with an optional
message as specified in OK_msg itself.
If OK_msg is an not NULL empty string, an OK packet is sent to the client
without any message.
If both error_msg and OK_msg are present, error_msg is returned.
This commit includes also few minor bugs, mostly related to typo that would
prevent online upgrade of mysql_query_rules from early release of 1.4.0 .
mysql_query_rules.multiplex=2 has a special meaning: do not disable multiplex for variables in THIS QUERY ONLY
Also fixed the upgrade of mysql_query_rules from 1.3
New command `PROXYSQL FLUSH CONFIGDB` allows to close and reopen configdb (`proxysql.db` on disk).
Note:
It is important that the (new) configdb is a valid one.
A misconfigured configdb will cause undefined behaviors.
* reduce build time by 25%;
* remove circular dependency on `cpp.h` and other headers.
* fixes issue with `my_global.h` from mariadbclient which
redefines `__attribute__`, that leads to broken STL and GCC
intrinsics if `<algorithm>` wasn't included prior including this header;
* fixes another issue with redefined `__attribute__` which leads to
`_conn_exchange_t` not being aligned to cache line size;
* removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize`
to be `pthread_dummy(0)` which led to stack size being never adjusted;
* add missing include guards to some headers;
* remove unused proxysql_hash.h header.
This commit is able to speed up LOAD MYSQL SERVERS TO RUNTIME up to 10x for very large mysql_servers table with tens of thousands of servers.
Main improvements:
* removed 1 unnecessary call to `generate_mysql_servers_table()`
* replaced 1 call to `generate_mysql_servers_table()` with UPDATE statements when necessary
* implemented new function `servers_add()` that replaces `server_add()` to perform bulk operation
* replaced calls to `sqlite3_exec()` with proper parameters bindings
* use of multirows INSERT statements (32 rows at the time)
Miror improvements:
* added new variable mysql-hostgroup_manager_verbose , 1 by default for backward compatibility and debugging
* in not debug mode, access to HGM db has no shared cache
* in not debug mode, tables in HGM have checks disabled
Further note:
* Once issue #797 is completed, `mysql_servers` in MyHGM can be simplified further
New variable admin-hash_passwords , default true
When set, passwords in mysql_users are automatically hashed in runtime
when running LOAD MYSQL USERS TO RUNTIME.
For backward compatibility, passwords in mysql_users in memory and disk aren't
automatically hashed. Although, they can be easily hashed running
SAVE MYSQL USERS TO MEMORY (and eventually SAVE MYSQL USERS TO DISK)
immediately after running LOAD MYSQL USERS TO RUNTIME
Command "LOAD SCHEDULER FROM CONFIG" is now implemented.
Config file can also be read with --initial, --reload, and when a db file is not present yet.
An example was added in src/proxysql.cfg
MySQL_STMT_Manager now stores 2 reference counters per statement: one for client and one for server
Statement client side are removed with STMT_CLOSE command is processed
Bytes sends and received are accounted globally and per server
The check if the configdb was not existing wasn't present anymore.
This probably happened when introducing the new LOAD FROM CONFIG commands.
Now the functionality is back.
Now Admin supports the configuration of mysql_replication_hostgroups from configfile.
That means that also LOAD MYSQL SERVERS FROM CONFIG is responsible for configuring mysql_replication_hostgroups.
This commit also allows comments in config file for `mysql_servers` and `mysql_replication_hostgroups` , related to #643
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
Added:
* scheduler.comment VARCHAR NOT NULL DEFAULT ''
Like the other tables, scheduler supports online upgrade from previous version (without such field) at startup.
Added:
* mysql_query_rules.comment VARCHAR
* mysql_replication_hostgroups.comment VARCHAR
* mysql_servers.comment VARCHAR NOT NULL DEFAULT ''
All these tables support online upgrade from previous version (without such fields) at startup.
Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping
Scheduler is being implemented as part of the Admin module.
Added table scheduler
Added table runtime_scheduler
Implemented commands:
- [LAD|SAVE] SCHEDULER [FROM|TO] [DISK|MEMORY|RUNTIME]
Issue #576
Added some inline timers
Reduced main loop sleep from 1 sec to 0.5 sec
Admin module is signaled to quit ASAP
MySQL Threads are signaled to quit ASAP
These two new columns will be used to:
- enable SSL on a per server basis
- define maximum latency for a server to be consider usable: this will be a key element for multi DC support
mysql_query_rules table is changing from version to version.
In order to provide online upgrade, a new function was added:
- ProxySQL_Admin::disk_upgrade_mysql_query_rules()
Expanded table `mysql_query_rules` to add `mirror_hostgroup`
Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup`
Added new variable `mirror_hostgroup` in `MySQL_Session`
The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session:
* when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor
* when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup`
Current limitations:
* queries larger than 15MB aren't mirrorred
* upgrade from 1.1 is still not possible it will wipe table mysql_query_rules
All save commands now make sure that the configs are first copied from
runtime to memory before theyre read and that they are copied from
memory to runtime after they are written
takes an action as the firt argument. put action requires two more
arguments: config value and config type. writes the config value to
consul on a key corresponding to the config type.
The follow states are disabled in Admin module is the backend:
- CHANGING_USER_SERVER
- CHANGING_SCHEMA
- CHANGING_CHARSET
- CHANGING_AUTOCOMMIT
- mysql-monitor_username is always added as a possible user : this allows MySQL_Monitor to monitor the backends
- admin-admin_credentials and admin-stats_credentials are always added as possible users
- mysql-monitor_username , admin-admin_credentials and stats-admin_credentials do not shows in mysql_users table (by design)
- the above users have a max_connections limit of 1000 (hardcoded)
Features added:
* added new global variable admin-read_only: it is possible to enable read_only changing this variable and running LOAD ADMIN VARIABLES TO RUNTIME
* admin interface now replies to SHOW GLOBAL VARIABLES LIKE 'read_only'
* added new command PROXYSQL READONLY to enable read only mode
* added new command PROXYSQL READWRITE to disable read only mode : this is the only way to disable read only mode while global_variables is not readable
Note that this doesn't prevent the execution of commands LOAD and SAVE : these commands need to be used with caution
Added 2 new functions to answer some of queries from mysqldump:
ProxySQL_Admin::generate_show_fields_from()
ProxySQL_Admin::generate_show_table_status()
ProxySQL_Admin() replies with an OK or an artificial resultset to most of the queries from mysqldump
Status variables implemented so far:
- Client_Connections_aborted
- Client_Connections_connected
- Client_Connections_created
- Questions
- Slow_queries
Also created alias command SHOW MYSQL STATUS
- typos in stats_mysql_query_digest_reset
- stats___mysql_query_digests() is executed on demand
- dump_mysql_collations() is moved after table creation
- Replaced some proxy_error() with daemon_log()
- glovars.proxy_restart_on_error is configurable via command line
- glovars.proxy_restart_delay is hardcoded to 1 second
- disabled listeners on telnet
- Option --admin-socket overwrite the default admin socket
- telnet_admin_ifaces disabled as currently not implemented
- telnet_stats_ifaces disabled as currently not implemented
- ProxySQL_Admin::get_variable() uses a safer version of strdup()
- default config file is, in order: proxysql.cnf , proxysql.cfg , /etc/proxysql.cnf , /etc/proxysql.cfg
- default datadir moved from /var/run/proxysql to current working directory
Minor bug fixes
Implemented LOAD MYSQL QUERY RULES FROM CONFIG via ProxySQL_Admin::Read_MySQL_Query_Rules_from_configfile()
Fixed bug in ProxySQL_Admin::save_mysql_query_rules_from_runtime()
Simplified the ping within MySQL_Thread
SQLite3_row::add_fields() to handle empty string
prot_status was not initialized in MySQL_Protocol
Session's default schema was always mysql_thread___default_schema , now fixed with correct value
default_schema is now read from config file
Dependencies:
- added libevent
- added mariadb-connector-c for non-blocking API
- added patch for mariadb-connector-c ( see https://mariadb.atlassian.net/browse/CONC-131 )
MySQL Monitoring:
- added table mysql_server_connect_log
- added table mysql_server_ping_log
- added access to admindb
- implemented monitoring using MariaDB client library non-blocking API and libevent, using MySQL_Monitor_State_Data to track state machine (experimental, and may change a lot in the near future)
- implemented MySQL_Monitor_Connection_Pool , a new connection pool only for MySQL Monitoring
- in MySQL_Thread added variable monitor_history to define the length of logs
- in MySQL_Thread added variables monitor_username and monitor_password to define monitoring credentials
- in MySQL_Thread added variable monitor_timer_cached to define if monitoring timers can be cached
- not completed feature: MySQL_Thread added variables monitor_query_variables, monitor_query_status, monitor_query_interval and monitor_query_timeout to define how global variables and global status are collected
Admin:
- added access to monitordb, to query MySQL_Monitor tables