mirror of https://github.com/sysown/proxysql
Add comprehensive documentation describing the protocol labels feature introduced in this branch to resolve metric collisions between MySQL and PostgreSQL prometheus metrics. Related: #5068, #5069pull/5361/head
Rene Cannao
2 months ago
parent
3ab964010d
commit
d90be4080a
@ -0,0 +1,380 @@
|
||||
# Prometheus Protocol Labels for PostgreSQL Metrics
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the features introduced in the `v3.0-5069` branch that resolve prometheus metric collisions between MySQL and PostgreSQL modules by adding `protocol` labels to distinguish metrics from different database protocols.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Issue #5068: Metric Collision Between MySQL and PostgreSQL
|
||||
|
||||
Prior to this branch, ProxySQL used identical metric names for both MySQL and PostgreSQL modules. When both modules tried to register prometheus metrics with the same name and labels, they would either:
|
||||
1. **Collide and overwrite** - causing incorrect metric values
|
||||
2. **Be disabled entirely** - PostgreSQL metrics were disabled to avoid collision
|
||||
|
||||
This resulted in users seeing confusing behavior where:
|
||||
- Prometheus showed `proxysql_client_connections_total{status="aborted"} 4` (MySQL metrics only)
|
||||
- Database showed `Client_Connections_aborted: 91233` (PostgreSQL stats)
|
||||
- The user was comparing MySQL prometheus metrics with PostgreSQL database stats
|
||||
|
||||
### Root Cause
|
||||
|
||||
The metric collision occurred because both modules used identical metric names with identical labels:
|
||||
|
||||
```cpp
|
||||
// MySQL_HostGroups_Manager.cpp
|
||||
"proxysql_client_connections_total", metric_tags { { "status", "created" } }
|
||||
|
||||
// PgSQL_HostGroups_Manager.cpp
|
||||
"proxysql_client_connections_total", metric_tags { { "status", "created" } } // IDENTICAL!
|
||||
```
|
||||
|
||||
When both instances called `BuildCounter().Register(...).Add(metric_tags)` with the same name and labels, they shared the same counter.
|
||||
|
||||
## Solution: Protocol Labels
|
||||
|
||||
The solution adds a `protocol` label to all prometheus metrics to distinguish MySQL and PostgreSQL sources:
|
||||
|
||||
```
|
||||
proxysql_client_connections_total{protocol="mysql",status="created"} 1000
|
||||
proxysql_client_connections_total{protocol="pgsql",status="created"} 500
|
||||
```
|
||||
|
||||
This follows prometheus best practices where metrics that represent different entities should be distinguished using labels.
|
||||
|
||||
## Changes Summary
|
||||
|
||||
| File | Lines Changed | Description |
|
||||
|------|--------------|-------------|
|
||||
| `lib/MySQL_HostGroups_Manager.cpp` | +25/-10 | Added `protocol="mysql"` labels to PgHGM metrics |
|
||||
| `lib/MySQL_Thread.cpp` | +340/-71 | Added `protocol="mysql"` labels to 65 MTH metrics |
|
||||
| `lib/PgSQL_HostGroups_Manager.cpp` | +25/-10 | Added `protocol="pgsql"` labels to PgHGM metrics |
|
||||
| `lib/PgSQL_Thread.cpp` | +310/-42 | Added `protocol="pgsql"` labels to 60 PTH metrics |
|
||||
| `lib/ProxySQL_Admin.cpp` | +18/-18 | Enabled GloPTH and GloPgQC metrics |
|
||||
| `lib/Query_Cache.cpp` | +133/-15 | Added protocol labels with compile-time type selection |
|
||||
| **Total** | **+784/-147** | **6 files modified** |
|
||||
|
||||
## Detailed Changes by Module
|
||||
|
||||
### 1. Host Groups Manager Metrics
|
||||
|
||||
#### MySQL_HostGroups_Manager.cpp
|
||||
|
||||
Added `protocol="mysql"` labels to the following metrics:
|
||||
- `proxysql_server_connections_total` (created/delayed/aborted)
|
||||
- `proxysql_client_connections_total` (created/aborted)
|
||||
- `proxysql_access_denied_wrong_password_total`
|
||||
- `proxysql_access_denied_max_connections_total`
|
||||
- `proxysql_access_denied_max_user_connections_total`
|
||||
- `proxysql_server_connections_connected`
|
||||
- `proxysql_client_connections_connected`
|
||||
|
||||
#### PgSQL_HostGroups_Manager.cpp
|
||||
|
||||
Added `protocol="pgsql"` labels to the same metrics for PostgreSQL.
|
||||
|
||||
### 2. Thread Handler Metrics
|
||||
|
||||
#### MySQL_Thread.cpp
|
||||
|
||||
Added `protocol="mysql"` labels to 65 thread handler metrics including:
|
||||
- `proxysql_queries_backends_bytes_total` (sent/received)
|
||||
- `proxysql_queries_frontends_bytes_total` (sent/received)
|
||||
- `proxysql_query_processor_time_seconds_total`
|
||||
- `proxysql_backend_query_time_seconds_total`
|
||||
- `proxysql_com_backend_stmt_total` (prepare/execute/close)
|
||||
- `proxysql_com_frontend_stmt_total` (prepare/execute/close)
|
||||
- `proxysql_questions_total`
|
||||
- `proxysql_slow_queries_total`
|
||||
- And 50+ more metrics
|
||||
|
||||
#### PgSQL_Thread.cpp
|
||||
|
||||
Added `protocol="pgsql"` labels to the same 60 metrics for PostgreSQL.
|
||||
|
||||
### 3. Query Cache Metrics
|
||||
|
||||
#### Query_Cache.cpp
|
||||
|
||||
This module presented a unique challenge because it uses a template class:
|
||||
```cpp
|
||||
template <typename QC_DERIVED>
|
||||
class Query_Cache : public /* ... */ { ... }
|
||||
```
|
||||
|
||||
Both `MySQL_Query_Cache` and `PgSQL_Query_Cache` inherit from this template and share the same metrics map.
|
||||
|
||||
**Solution:** Created two separate metric maps:
|
||||
- `qc_metrics_map_mysql` - with `{ "protocol", "mysql" }` labels
|
||||
- `qc_metrics_map_pgsql` - with `{ "protocol", "pgsql" }` labels
|
||||
|
||||
The constructor uses `if constexpr (std::is_same_v<QC_DERIVED, MySQL_Query_Cache>)` to select the appropriate map at compile time:
|
||||
|
||||
```cpp
|
||||
template <typename QC_DERIVED>
|
||||
Query_Cache<QC_DERIVED>::Query_Cache() {
|
||||
// ... existing code ...
|
||||
|
||||
// Select metrics map based on derived type
|
||||
if constexpr (std::is_same_v<QC_DERIVED, MySQL_Query_Cache>) {
|
||||
init_prometheus_counter_array<qc_metrics_map_idx, p_qc_counter>(
|
||||
qc_metrics_map_mysql, this->metrics.p_counter_array);
|
||||
init_prometheus_gauge_array<qc_metrics_map_idx, p_qc_gauge>(
|
||||
qc_metrics_map_mysql, this->metrics.p_gauge_array);
|
||||
} else {
|
||||
init_prometheus_counter_array<qc_metrics_map_idx, p_qc_counter>(
|
||||
qc_metrics_map_pgsql, this->metrics.p_counter_array);
|
||||
init_prometheus_gauge_array<qc_metrics_map_idx, p_qc_gauge>(
|
||||
qc_metrics_map_pgsql, this->metrics.p_gauge_array);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Added protocol labels to 8 Query Cache metrics:
|
||||
- `proxysql_query_cache_count_get_total` (status=err/ok)
|
||||
- `proxysql_query_cache_count_set_total`
|
||||
- `proxysql_query_cache_bytes_total` (op=written/read)
|
||||
- `proxysql_query_cache_purged_total`
|
||||
- `proxysql_query_cache_entries_total`
|
||||
- `proxysql_query_cache_memory_bytes`
|
||||
|
||||
### 4. Metrics Update Coordination
|
||||
|
||||
#### ProxySQL_Admin.cpp
|
||||
|
||||
Enabled the following metrics updates in `update_modules_metrics()`:
|
||||
|
||||
1. **PgHGM** (PostgreSQL host groups manager) - Enabled by PR #5069
|
||||
2. **GloPTH** (PostgreSQL threads handler) - Newly enabled
|
||||
3. **GloPgQC** (PostgreSQL query cache) - Newly enabled
|
||||
|
||||
```cpp
|
||||
void update_modules_metrics() {
|
||||
// ... existing MySQL metrics updates ...
|
||||
|
||||
// PostgreSQL host groups manager metrics
|
||||
if (PgHGM) {
|
||||
PgHGM->p_update_metrics();
|
||||
}
|
||||
// PostgreSQL threads handler metrics
|
||||
if (GloPTH) {
|
||||
GloPTH->p_update_metrics();
|
||||
}
|
||||
// PostgreSQL query cache metrics
|
||||
if (GloPgQC) {
|
||||
GloPgQC->p_update_metrics();
|
||||
}
|
||||
// ... rest of metrics updates ...
|
||||
}
|
||||
```
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Type Traits for Compile-Time Selection
|
||||
|
||||
The Query Cache implementation uses C++17 type traits for zero-overhead compile-time dispatch:
|
||||
|
||||
```cpp
|
||||
#include <type_traits>
|
||||
|
||||
// Returns true at compile time when QC_DERIVED is MySQL_Query_Cache
|
||||
if constexpr (std::is_same_v<QC_DERIVED, MySQL_Query_Cache>) {
|
||||
// Use MySQL metrics map
|
||||
} else {
|
||||
// Use PostgreSQL metrics map
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Zero runtime overhead - the `if constexpr` branch is resolved at compile time
|
||||
- No performance penalty
|
||||
- Type-safe - the compiler validates the logic
|
||||
|
||||
### Metric Label Architecture
|
||||
|
||||
All metrics now follow this label structure:
|
||||
|
||||
**Before:**
|
||||
```cpp
|
||||
metric_tags { { "status", "created" } }
|
||||
```
|
||||
|
||||
**After:**
|
||||
```cpp
|
||||
metric_tags { { "status", "created" }, { "protocol", "mysql" } }
|
||||
```
|
||||
|
||||
For metrics without existing labels:
|
||||
```cpp
|
||||
metric_tags { { "protocol", "mysql" } }
|
||||
```
|
||||
|
||||
## Prometheus Metrics Examples
|
||||
|
||||
### Thread Handler Metrics
|
||||
|
||||
```bash
|
||||
# MySQL Thread Handler Metrics
|
||||
proxysql_queries_backends_bytes_total{protocol="mysql",traffic_flow="sent"} 1234567
|
||||
proxysql_queries_backends_bytes_total{protocol="mysql",traffic_flow="received"} 2345678
|
||||
proxysql_com_backend_stmt_total{protocol="mysql",op="prepare"} 100
|
||||
proxysql_com_backend_stmt_total{protocol="mysql",op="execute"} 1000
|
||||
proxysql_questions_total{protocol="mysql"} 50000
|
||||
|
||||
# PostgreSQL Thread Handler Metrics
|
||||
proxysql_queries_backends_bytes_total{protocol="pgsql",traffic_flow="sent"} 345678
|
||||
proxysql_queries_backends_bytes_total{protocol="pgsql",traffic_flow="received"} 456789
|
||||
proxysql_com_backend_stmt_total{protocol="pgsql",op="prepare"} 50
|
||||
proxysql_com_backend_stmt_total{protocol="pgsql",op="execute"} 500
|
||||
proxysql_questions_total{protocol="pgsql"} 25000
|
||||
```
|
||||
|
||||
### Host Groups Manager Metrics
|
||||
|
||||
```bash
|
||||
# MySQL Host Groups Manager Metrics
|
||||
proxysql_client_connections_total{protocol="mysql",status="created"} 1000000
|
||||
proxysql_client_connections_total{protocol="mysql",status="aborted"} 5000
|
||||
proxysql_server_connections_total{protocol="mysql",status="created"} 200000
|
||||
proxysql_access_denied_wrong_password_total{protocol="mysql"} 100
|
||||
proxysql_server_connections_connected{protocol="mysql"} 500
|
||||
|
||||
# PostgreSQL Host Groups Manager Metrics
|
||||
proxysql_client_connections_total{protocol="pgsql",status="created"} 500000
|
||||
proxysql_client_connections_total{protocol="pgsql",status="aborted"} 2500
|
||||
proxysql_server_connections_total{protocol="pgsql",status="created"} 100000
|
||||
proxysql_access_denied_wrong_password_total{protocol="pgsql"} 50
|
||||
proxysql_server_connections_connected{protocol="pgsql"} 250
|
||||
```
|
||||
|
||||
### Query Cache Metrics
|
||||
|
||||
```bash
|
||||
# MySQL Query Cache Metrics
|
||||
proxysql_query_cache_count_get_total{protocol="mysql",status="err"} 100
|
||||
proxysql_query_cache_count_get_total{protocol="mysql",status="ok"} 50000
|
||||
proxysql_query_cache_count_set_total{protocol="mysql"} 25000
|
||||
proxysql_query_cache_bytes_total{protocol="mysql",op="read"} 104857600
|
||||
proxysql_query_cache_bytes_total{protocol="mysql",op="written"} 52428800
|
||||
proxysql_query_cache_entries_total{protocol="mysql"} 1000
|
||||
proxysql_query_cache_memory_bytes{protocol="mysql"} 2097152
|
||||
|
||||
# PostgreSQL Query Cache Metrics
|
||||
proxysql_query_cache_count_get_total{protocol="pgsql",status="err"} 50
|
||||
proxysql_query_cache_count_get_total{protocol="pgsql",status="ok"} 25000
|
||||
proxysql_query_cache_count_set_total{protocol="pgsql"} 12500
|
||||
proxysql_query_cache_bytes_total{protocol="pgsql",op="read"} 52428800
|
||||
proxysql_query_cache_bytes_total{protocol="pgsql",op="written"} 26214400
|
||||
proxysql_query_cache_entries_total{protocol="pgsql"} 500
|
||||
proxysql_query_cache_memory_bytes{protocol="pgsql"} 1048576
|
||||
```
|
||||
|
||||
## Related Issues and Pull Requests
|
||||
|
||||
- **Issue #5068**: Original issue reporting the metric discrepancy
|
||||
- **PR #5069**: Initial PR that added protocol labels to Host Groups Manager metrics
|
||||
- **Branch v3.0-5069**: This branch extends PR #5069 by adding protocol labels to all MySQL and PostgreSQL prometheus metrics
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
None. The changes only add new labels to existing metrics. Existing prometheus queries that don't filter by `protocol` will continue to work but will now see duplicate time series.
|
||||
|
||||
### Prometheus Query Impact
|
||||
|
||||
**Before:** Queries would return a single time series per metric:
|
||||
```promql
|
||||
proxysql_client_connections_total{status="created"}
|
||||
```
|
||||
|
||||
**After:** Queries will return multiple time series (one per protocol):
|
||||
```promql
|
||||
proxysql_client_connections_total{status="created"}
|
||||
```
|
||||
|
||||
To maintain existing behavior, users should add a protocol filter:
|
||||
```promql
|
||||
proxysql_client_connections_total{status="created",protocol="mysql"}
|
||||
```
|
||||
|
||||
### Recommended Prometheus Queries
|
||||
|
||||
**Total connections across all protocols:**
|
||||
```promql
|
||||
sum(proxysql_client_connections_total{status="created"})
|
||||
```
|
||||
|
||||
**MySQL-only metrics:**
|
||||
```promql
|
||||
proxysql_client_connections_total{status="created",protocol="mysql"}
|
||||
```
|
||||
|
||||
**PostgreSQL-only metrics:**
|
||||
```promql
|
||||
proxysql_client_connections_total{status="created",protocol="pgsql"}
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
To verify the implementation:
|
||||
|
||||
1. **Build ProxySQL:**
|
||||
```bash
|
||||
make clean && make -j4
|
||||
```
|
||||
|
||||
2. **Check prometheus metrics:**
|
||||
```bash
|
||||
curl localhost:6070/metrics | grep -E "protocol=.*(mysql|pgsql)"
|
||||
```
|
||||
|
||||
3. **Verify separate metrics exist:**
|
||||
```bash
|
||||
curl localhost:6070/metrics | grep "proxysql_client_connections_total"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
proxysql_client_connections_total{protocol="mysql",status="aborted"} X
|
||||
proxysql_client_connections_total{protocol="mysql",status="created"} Y
|
||||
proxysql_client_connections_total{protocol="pgsql",status="aborted"} Z
|
||||
proxysql_client_connections_total{protocol="pgsql",status="created"} W
|
||||
```
|
||||
|
||||
## Implementation Commits
|
||||
|
||||
1. **fa35bda62** - Merge pull request #5069
|
||||
- Added protocol labels to MySQL_HostGroups_Manager
|
||||
- Added protocol labels to PgSQL_HostGroups_Manager
|
||||
- Enabled PgHGM metrics export
|
||||
|
||||
2. **2b44aaa58** - Add protocol labels for shared metrics between mysql and psql
|
||||
- Initial implementation of protocol label approach
|
||||
|
||||
3. **949eda1cc** - Generate postgres metrics in addition to mysql metrics
|
||||
- Enabled PgHGM metrics export
|
||||
|
||||
4. **778e01174** - Add protocol labels to Thread Handler metrics and enable PostgreSQL metrics
|
||||
- Added protocol labels to MySQL_Thread.cpp (65 metrics)
|
||||
- Added protocol labels to PgSQL_Thread.cpp (60 metrics)
|
||||
- Enabled GloPTH metrics export
|
||||
|
||||
5. **3ab964010** - Add protocol labels to Query Cache metrics and enable PostgreSQL QC metrics
|
||||
- Added `#include <type_traits>`
|
||||
- Created qc_metrics_map_mysql with protocol="mysql" labels
|
||||
- Created qc_metrics_map_pgsql with protocol="pgsql" labels
|
||||
- Modified Query_Cache constructor with if constexpr type selection
|
||||
- Enabled GloPgQC metrics export
|
||||
|
||||
## Summary
|
||||
|
||||
This implementation fully resolves Issue #5068 by adding protocol labels to all prometheus metrics, allowing MySQL and PostgreSQL modules to coexist without metric collisions. The solution:
|
||||
|
||||
- ✅ **Follows prometheus best practices** - using labels to distinguish metric sources
|
||||
- ✅ **Maintains backward compatibility** - existing queries continue to work
|
||||
- ✅ **Has zero runtime overhead** - uses compile-time type trait selection
|
||||
- ✅ **Is maintainable** - clear separation of MySQL and PostgreSQL metrics
|
||||
- ✅ **Completes PR #5069** - extends the initial work to cover all modules
|
||||
|
||||
Users can now monitor MySQL and PostgreSQL metrics independently using protocol filters, providing complete visibility into both database protocols.
|
||||
Loading…
Reference in new issue