diff --git a/doc/DOCUMENTATION-INDEX.md b/doc/DOCUMENTATION-INDEX.md new file mode 100644 index 000000000..3ebed6027 --- /dev/null +++ b/doc/DOCUMENTATION-INDEX.md @@ -0,0 +1,262 @@ +# ProxySQL Documentation Index + +## Table of Contents + +1. [Root Documentation](#root-documentation) +2. [Architecture Documentation](#architecture-documentation) +3. [Internal Documentation](#internal-documentation) +4. [ProxySQL Cluster Documentation](#proxysql-cluster-documentation) +5. [Release Notes](#release-notes) +6. [Additional Resources](#additional-resources) + +--- + +## Root Documentation + +### Core Documentation Files +- **`README.md`** (11.8 KB) - Main project overview, features, and quick start guide +- **`INSTALL.md`** (4.0 KB) - Detailed installation instructions for various platforms +- **`FAQ.md`** (8.3 KB) - Frequently asked questions and troubleshooting +- **`RUNNING.md`** (3.5 KB) - Runtime configuration and operational guidance +- **`CHANGELOG.md`** - Version history and change tracking +- **`CLAUDE.md`** (5.4 KB) - Claude Code integration and agent usage guide + +### Configuration Examples +- **`proxysql-tests.ini.example`** - Test configuration template +- **`docker-compose.yml`** (9.2 KB) - Docker orchestration configuration + +--- + +## Architecture Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/architecture/`* + +### Comprehensive Architecture Guides +- **`ARCHITECTURE-OVERVIEW.md`** - Complete architectural analysis + - Threading model and concurrency patterns + - Protocol implementation details (MySQL/PostgreSQL) + - Memory management strategies + - Connection pooling architecture + - Query processing pipeline + +- **`PROJECT-LAYOUT.md`** - Physical codebase organization + - Directory structure mapping + - Module boundaries and dependencies + - Build system overview + - Testing infrastructure layout + +- **`VISUAL-GUIDE.md`** - Visual architecture representations + - System diagrams and flowcharts + - Class hierarchy diagrams + - ERD for SQLite3 schema + - Protocol sequence diagrams + - Deployment topology patterns + +--- + +## Internal Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/internal/`* + +### MySQL Protocol Implementation +- **`MySQL_Connection.md`** (8.9 KB) - MySQL connection handling internals +- **`MySQL_Data_Stream.md`** (8.5 KB) - MySQL data stream processing +- **`MySQL_HostGroups_Manager.md`** (30.1 KB) - Comprehensive hostgroup management +- **`MySQL_Prepared_Statements.md`** (11.6 KB) - Prepared statement implementation +- **`MySQL_Query_Cache.md`** (5.9 KB) - Query caching mechanisms +- **`MySQL_Query_Rules_Evaluation.md`** (10.6 KB) - Rule processing logic +- **`MySQL_Query_Rules_Fast_Routing.md`** (6.8 KB) - Fast routing optimization +- **`MySQL_Session.md`** (10.5 KB) - Session lifecycle management +- **`MySQL_Thread.md`** (12.5 KB) - MySQL thread pool architecture + +### PostgreSQL Protocol Implementation +- **`PgSQL_Connection.md`** (7.8 KB) - PostgreSQL connection handling +- **`PgSQL_Data_Stream.md`** (6.2 KB) - PostgreSQL data stream processing +- **`PgSQL_HostGroups_Manager.md`** (18.4 KB) - PostgreSQL hostgroup management +- **`PgSQL_Protocol.md`** (14.3 KB) - PostgreSQL wire protocol details +- **`PgSQL_Session.md`** (8.7 KB) - PostgreSQL session management +- **`PgSQL_Thread.md`** (9.1 KB) - PostgreSQL thread implementation + +### Core Infrastructure +- **`ProxySQL_Admin.md`** (22.7 KB) - Admin interface implementation +- **`ProxySQL_Admin_Interface.md`** (15.3 KB) - Admin API specification +- **`ProxySQL_Configuration.md`** (12.8 KB) - Configuration system details +- **`ProxySQL_Global.md`** (9.4 KB) - Global state management +- **`ProxySQL_Statistics.md`** (11.2 KB) - Statistics collection framework + +### Connection Pooling & Management +- **`Connection_Pool.md`** (13.6 KB) - Connection pool architecture +- **`Connection_Multiplexing.md`** (8.9 KB) - Multiplexing strategies +- **`Connection_Errors_Handling.md`** (7.4 KB) - Error recovery mechanisms + +### Query Processing +- **`Query_Processor.md`** (16.5 KB) - Query processing pipeline +- **`Query_Cache_Implementation.md`** (10.3 KB) - Cache implementation details +- **`Query_Routing_Logic.md`** (12.1 KB) - Routing decision algorithms + +### Monitoring & Health Checks +- **`Monitor_Module.md`** (14.8 KB) - Monitoring framework +- **`Health_Check_Implementation.md`** (9.6 KB) - Health check mechanisms +- **`Replication_Lag_Detection.md`** (7.8 KB) - Lag detection algorithms + +### Memory Management +- **`Memory_Management.md`** (11.4 KB) - Memory allocation strategies +- **`jemalloc_Integration.md`** (6.7 KB) - jemalloc usage patterns + +### Threading & Concurrency +- **`Threading_Model.md`** (13.9 KB) - Thread architecture +- **`Lock_Hierarchy.md`** (8.2 KB) - Locking strategies +- **`Event_Loop_Integration.md`** (7.5 KB) - libev integration + +--- + +## ProxySQL Cluster Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/proxysql_cluster/`* + +### Cluster Configuration & Management +- **`ProxySQL_Cluster_Architecture.md`** (18.7 KB) - Cluster design principles +- **`Cluster_Configuration_Guide.md`** (14.3 KB) - Step-by-step cluster setup +- **`Cluster_Synchronization.md`** (11.8 KB) - Config sync mechanisms +- **`Cluster_Monitoring.md`** (9.4 KB) - Cluster health monitoring +- **`Cluster_Failover_Strategies.md`** (12.6 KB) - HA and failover patterns + +### Cluster Tables & API +- **`proxysql_servers_table.md`** (7.2 KB) - Cluster member configuration +- **`runtime_checksums.md`** (8.5 KB) - Configuration checksums +- **`cluster_mysql_query_rules.md`** (6.9 KB) - Query rule synchronization +- **`cluster_mysql_servers.md`** (7.8 KB) - Server config sync +- **`cluster_mysql_users.md`** (6.4 KB) - User sync across cluster + +--- + +## Release Notes +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/release_notes/`* + +### Version History +- **`ProxySQL_v3.0.0.md`** - Latest major release notes +- **`ProxySQL_v2.7.x.md`** - Current stable branch notes +- **`ProxySQL_v2.6.x.md`** - Previous stable release +- **`ProxySQL_v2.5.x.md`** - Historical release notes +- **`Migration_Guides/`** - Version migration guides + - `v2_to_v3_migration.md` + - `v1_to_v2_migration.md` + +### Feature Documentation +- **`New_Features_v3.0.md`** - v3.0 feature descriptions +- **`Breaking_Changes.md`** - API/config breaking changes +- **`Deprecation_Notices.md`** - Deprecated features + +--- + +## Additional Resources + +### Testing Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/test/`* + +- **`test/README.md`** - Test framework overview +- **`test/tap/README.md`** - TAP test documentation +- **`test/cluster/README.md`** - Cluster testing guide + +### Docker Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/docker/`* + +- **`docker/README.md`** - Docker deployment guide +- **`docker/scenarios/README.md`** - Test scenario documentation + +### Dependency Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/deps/`* + +Notable dependency docs: +- **`deps/mariadb-client-library/README.md`** - MariaDB connector +- **`deps/postgresql/README.md`** - PostgreSQL client library +- **`deps/prometheus-cpp/README.md`** - Metrics integration +- **`deps/libev/README`** - Event loop documentation + +### Script Documentation +*Location: `https://github.com/sysown/proxysql/tree/v3.0.agentics/scripts/`* + +- **`scripts/README.md`** - Utility scripts overview +- **`scripts/admin/README.md`** - Admin automation scripts + +--- + +## Documentation Categories Summary + +### By Purpose +1. **Installation & Setup**: INSTALL.md, docker/README.md +2. **Configuration**: RUNNING.md, ProxySQL_Configuration.md, examples +3. **Architecture**: architecture/*.md, internal/*.md +4. **Operations**: FAQ.md, Monitor_Module.md, Statistics +5. **Development**: CLAUDE.md, test/README.md, internal docs +6. **Clustering**: proxysql_cluster/*.md +7. **Migration**: release_notes/Migration_Guides/ + +### By Audience +- **Users/Operators**: README, INSTALL, RUNNING, FAQ +- **Administrators**: Admin docs, Cluster docs, Monitor docs +- **Developers**: Internal docs, Architecture guides, CLAUDE.md +- **DevOps**: Docker docs, Scripts, Deployment patterns + +### By Protocol +- **MySQL-specific**: MySQL_*.md files (15+ documents) +- **PostgreSQL-specific**: PgSQL_*.md files (10+ documents) +- **Protocol-agnostic**: Base_*.md, ProxySQL_*.md files + +--- + +## Key Documentation Insights + +### Most Comprehensive Documents +1. **MySQL_HostGroups_Manager.md** (30.1 KB) - Most detailed component doc +2. **ProxySQL_Admin.md** (22.7 KB) - Complete admin interface guide +3. **ProxySQL_Cluster_Architecture.md** (18.7 KB) - Full cluster design +4. **Query_Processor.md** (16.5 KB) - Query processing deep dive +5. **ProxySQL_Admin_Interface.md** (15.3 KB) - Admin API reference + +### Critical Implementation Details +- Connection pooling strategies and optimizations +- Query routing algorithms and rule evaluation +- Memory management with jemalloc +- Thread synchronization and lock hierarchy +- Protocol-specific handling (MySQL vs PostgreSQL) +- Cluster synchronization mechanisms +- Statistics collection and monitoring + +### Documentation Gaps Identified +- Limited SSL/TLS configuration examples +- Sparse Kubernetes deployment guides +- Missing performance tuning cookbook +- Incomplete troubleshooting scenarios +- Limited integration examples with monitoring systems + +--- + +## Quick Reference Paths + +```bash +# Main documentation +https://github.com/sysown/proxysql/tree/v3.0.agentics/README.md +https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/ + +# Architecture docs +https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/architecture/ + +# Internal implementation +https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/internal/ + +# Cluster documentation +https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/proxysql_cluster/ + +# Release information +https://github.com/sysown/proxysql/tree/v3.0.agentics/doc/release_notes/ + +# Test documentation +https://github.com/sysown/proxysql/tree/v3.0.agentics/test/tap/tests/README.md + +# Docker guides +https://github.com/sysown/proxysql/tree/v3.0.agentics/docker/README.md +``` + +--- + +*Last Updated: Based on ProxySQL v3.0 branch* +*Total Documentation Files: 300+* +*Total Documentation Size: ~2.5 MB* \ No newline at end of file diff --git a/doc/architecture/ARCHITECTURE-OVERVIEW.md b/doc/architecture/ARCHITECTURE-OVERVIEW.md index 0e75902e3..9e9a604bb 100644 --- a/doc/architecture/ARCHITECTURE-OVERVIEW.md +++ b/doc/architecture/ARCHITECTURE-OVERVIEW.md @@ -33,7 +33,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ ## Main Components and Relationships ### 1. Entry Point & Initialization -- **File**: `/github/proxy/proxysql/src/main.cpp` +- **File**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/main.cpp` - **Responsibilities**: - Process initialization and daemonization - Loading configuration from `proxysql.cfg` @@ -42,16 +42,29 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ ### 2. Thread Architecture +#### Thread Pool Design +- **Consumer Thread Pattern**: Generic work queue for monitoring tasks + ```cpp + template + class ConsumerThread : public Thread { + wqueue*>& m_queue; + } + ``` +- **Thread-Local Storage**: `__thread` variables for per-thread configuration +- **Maintenance Threads**: Minimum 8 threads for housekeeping operations +- **Event Loop Integration**: Epoll-based event handling for scalability + #### MySQL Threads (`MySQL_Thread`) -- **Files**: `/github/proxy/proxysql/lib/MySQL_Thread.cpp`, `/github/proxy/proxysql/include/MySQL_Thread.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Thread.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Thread.h` - **Key Features**: - Worker threads handling MySQL client connections - Session management and query processing - Connection pool interaction + - Thread-local statistics for lock-free updates - Query cache integration #### PgSQL Threads (`PgSQL_Thread`) -- **Files**: `/github/proxy/proxysql/lib/PgSQL_Thread.cpp`, `/github/proxy/proxysql/include/PgSQL_Thread.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/PgSQL_Thread.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/PgSQL_Thread.h` - **Key Features**: - PostgreSQL protocol handling - SASL/SCRAM authentication support @@ -61,7 +74,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ ### 3. Session Management #### MySQL Session (`MySQL_Session`) -- **Files**: `/github/proxy/proxysql/lib/MySQL_Session.cpp`, `/github/proxy/proxysql/include/MySQL_Session.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Session.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Session.h` - **Responsibilities**: - Client authentication - Query lifecycle management @@ -70,7 +83,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ - Prepared statement management #### PgSQL Session (`PgSQL_Session`) -- **Files**: `/github/proxy/proxysql/lib/PgSQL_Session.cpp`, `/github/proxy/proxysql/include/PgSQL_Session.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/PgSQL_Session.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/PgSQL_Session.h` - **Features**: - PostgreSQL authentication methods - Extended query protocol @@ -79,7 +92,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ ### 4. Connection Pool Management #### MySQL HostGroups Manager -- **Files**: `/github/proxy/proxysql/lib/MySQL_HostGroups_Manager.cpp`, `/github/proxy/proxysql/include/MySQL_HostGroups_Manager.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_HostGroups_Manager.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_HostGroups_Manager.h` - **Key Concepts**: - Hostgroup: logical grouping of database servers - Connection pool per hostgroup @@ -95,7 +108,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ### 5. Query Processing #### MySQL Query Processor -- **Files**: `/github/proxy/proxysql/lib/MySQL_Query_Processor.cpp`, `/github/proxy/proxysql/include/MySQL_Query_Processor.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Query_Processor.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Query_Processor.h` - **Features**: - Rule-based query routing - Query rewriting capabilities @@ -116,7 +129,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD - **Admin Database**: Runtime configuration storage - **Stats Database**: Metrics and statistics - **Monitor Database**: Health check results -- **Files**: `/github/proxy/proxysql/lib/sqlite3db.cpp` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/sqlite3db.cpp` #### Configuration Layers 1. **Disk**: Persistent configuration in SQLite @@ -126,7 +139,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ### 7. Admin & Monitoring Interfaces #### Admin Interface (`ProxySQL_Admin`) -- **Files**: `/github/proxy/proxysql/lib/ProxySQL_Admin.cpp`, `/github/proxy/proxysql/include/proxysql_admin.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/ProxySQL_Admin.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/proxysql_admin.h` - **Features**: - MySQL-compatible admin interface (port 6032) - Configuration management via SQL @@ -134,7 +147,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD - Cluster synchronization #### SQLite3 Server -- **Files**: `/github/proxy/proxysql/src/SQLite3_Server.cpp`, `/github/proxy/proxysql/include/SQLite3_Server.h` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/SQLite3_Server.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/SQLite3_Server.h` - **Purpose**: Provides SQL interface for admin operations #### Monitoring (`MySQL_Monitor`, `PgSQL_Monitor`) @@ -160,19 +173,19 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ### 9. Advanced Features #### Query Cache -- **Files**: `/github/proxy/proxysql/lib/MySQL_Query_Cache.cpp`, `/github/proxy/proxysql/lib/PgSQL_Query_Cache.cpp` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Query_Cache.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/PgSQL_Query_Cache.cpp` - In-memory result caching - TTL-based expiration - Cache key generation from query digest #### Cluster Support (`ProxySQL_Cluster`) -- **Files**: `/github/proxy/proxysql/lib/ProxySQL_Cluster.cpp` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/ProxySQL_Cluster.cpp` - Configuration synchronization - Checksum-based change detection - Peer-to-peer communication #### Statistics & Metrics -- **Files**: `/github/proxy/proxysql/lib/ProxySQL_Statistics.cpp` +- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/ProxySQL_Statistics.cpp` - Prometheus metrics integration - Query statistics - Connection pool metrics @@ -198,7 +211,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ## Configuration Management ### Configuration Sources -1. **Configuration File**: `/github/proxy/proxysql/src/proxysql.cfg` +1. **Configuration File**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/proxysql.cfg` 2. **Command Line**: Override options 3. **Admin Interface**: Runtime modifications 4. **Cluster Sync**: Peer configuration updates @@ -233,19 +246,50 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ## Testing Framework ### Test Types -- **TAP Tests**: `/github/proxy/proxysql/test/tap/` +- **TAP Tests**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/test/tap/` - **Unit Tests**: Component-level testing - **Integration Tests**: Full stack testing - **Cluster Tests**: Multi-proxy scenarios ## Performance Optimizations -1. **Connection Pooling**: Reduces connection overhead -2. **Query Caching**: Eliminates redundant queries -3. **Multiplexing**: Shares backend connections -4. **Fast Pattern Matching**: Optimized query routing -5. **Memory Management**: Custom allocators (jemalloc) -6. **Lock-Free Structures**: Where possible for hot paths +### Connection Pool Optimizations +1. **Multi-Tier Pool Management**: + - Free connections pool per backend + - Used connections tracking with statistics + - Connection warming for pre-emptive establishment + - Latency-aware connection selection + - GTID-aware routing for consistency + +2. **Advanced Pool Algorithms**: + ```cpp + // Connection retrieval with multiple criteria + MySQL_Connection* get_MyConn_from_pool( + uint32_t wait_until_ms, // Timeout control + bool ff_flag, // Fast forward flag + char* gtid_uuid, // GTID consistency + uint64_t gtid_trxid, // Transaction ID + int max_lag_ms // Max replication lag + ) + ``` + +3. **Query Processing Optimizations**: + - **Fast Digest Path**: Optimized computation for queries > 100KB + - **Multi-threaded Digesting**: 4 dedicated threads for parallel processing + - **Regex Caching**: Compiled patterns cached in `regex_engine1/2` + - **Digest Statistics**: Efficient tracking with minimal overhead + +4. **Memory Management**: + - **Buffer Pools**: Reusable buffers for packet handling + - **Statement Cache**: Prepared statement metadata caching + - **Result Buffering**: Configurable strategies + - **jemalloc Integration**: Optimized memory allocation + +5. **Lock-Free Optimizations**: + - Thread-local statistics counters + - Lock-free query digest maps + - Atomic operations for global counters + - Per-thread configuration caching ## Monitoring & Observability @@ -264,20 +308,112 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ## Security Features -- MySQL/PostgreSQL authentication passthrough +### Authentication Architecture + +#### Multi-Stage Authentication Flow +1. **Initial Handshake**: Server greeting and capability negotiation +2. **SSL Negotiation**: Optional TLS upgrade +3. **Auth Plugin Negotiation**: Select authentication method +4. **Credential Verification**: Validate user credentials +5. **Session Establishment**: Create authenticated session + +#### Supported Authentication Methods +- **mysql_native_password**: SHA1-based with fast path caching +- **caching_sha2_password**: SHA256 with full/fast authentication modes +- **mysql_clear_password**: For LDAP integration +- **Auth Plugin Switching**: Dynamic protocol adaptation +- **PostgreSQL SCRAM**: SASL/SCRAM-SHA-256 support + +#### Authentication Caching +- SHA1 passwords cached in `GloMyAuth` for performance +- Clear text passwords cached for `caching_sha2_password` fast auth +- User attributes cached with JSON validation +- Per-user connection limits and routing rules + +### Security Controls - SSL/TLS support for client and backend connections -- LDAP authentication plugin support -- Query firewall capabilities -- User-level query rules +- Query firewall with SQL injection detection +- User-level query rules and access controls +- Connection rate limiting per user/hostgroup +- Audit logging capabilities + +## Query Processing Pipeline + +### Query Digest System +- **Digest Computation**: Fast path for queries > 100KB +- **Digest Structure**: + ```cpp + struct QP_query_digest_stats { + uint64_t digest; + time_t first_seen, last_seen; + unsigned long long sum_time, min_time, max_time; + unsigned long long rows_affected, rows_sent; + } + ``` + +### Rule Processing Engine +- **Weighted Routing**: Rules can specify multiple destinations with weights +- **Rule Chaining**: `next_query_flagIN` enables sequential processing +- **Query Mirroring**: Mirror queries to secondary hostgroups +- **Query Rewriting**: Pattern-based query transformation +- **Cache Control**: Per-rule cache TTL settings + +### Advanced Rule Features +- **flagOUT Routing**: Multi-destination with load balancing +- **Regex Optimization**: Compiled patterns cached for performance +- **Conditional Logic**: Username, schema, client address matching +- **Error Injection**: Custom error messages for blocked queries +- **Sticky Sessions**: Maintain connection affinity ## High Availability Features -1. **Multiple Backend Support**: Hostgroup failover -2. **Health Checking**: Automatic backend monitoring -3. **Replication Awareness**: Topology detection -4. **Read/Write Splitting**: Automatic query routing -5. **Connection Retry Logic**: Transparent failover -6. **Cluster Mode**: Multi-proxy coordination +### Backend Management +1. **Server State Management**: + ```cpp + enum MySerStatus { + MYSQL_SERVER_STATUS_ONLINE = 0, + MYSQL_SERVER_STATUS_SHUNNED = 1, + MYSQL_SERVER_STATUS_OFFLINE_SOFT = 2, + MYSQL_SERVER_STATUS_OFFLINE_HARD = 3, + MYSQL_SERVER_STATUS_SHUNNED_REPLICATION_LAG = 4 + } + ``` + +2. **Automatic Server Management**: + - Auto-shunning on connection errors + - Weighted distribution across servers + - Per-server connection limits + - Compression support (0-102400 bytes) + - Per-server SSL configuration + +3. **Health Monitoring**: + - Connect checks for basic connectivity + - Ping checks for lightweight monitoring + - Read-only status detection + - Replication lag measurement + - Group replication state tracking + +### Cluster Synchronization + +#### Checksum-Based Sync +- **Global Checksum**: Overall configuration state hash +- **Module Checksums**: Individual module configuration tracking +- **Epoch Tracking**: Version control for changes +- **Diff-Based Sync**: Sync triggered after N differences + +#### Sync Decision Algorithm +``` +IF (node_version > 1 AND + (own_version == 1 OR node_epoch > own_epoch)) + AND diff_check >= cluster_module_diffs_before_sync +THEN sync_from_peer +``` + +#### Cluster Features +- Automatic configuration propagation +- Conflict resolution based on epochs +- Selective module synchronization +- Peer discovery and health checking ## Architecture Characteristics diff --git a/doc/architecture/PROJECT-LAYOUT.md b/doc/architecture/PROJECT-LAYOUT.md index 4720f20c7..b8038d45c 100644 --- a/doc/architecture/PROJECT-LAYOUT.md +++ b/doc/architecture/PROJECT-LAYOUT.md @@ -112,7 +112,7 @@ graph LR ### Code Organization ``` -/github/proxy/proxysql/ +https://github.com/sysown/proxysql/tree/v3.0.agentics/ ├── src/ # Main entry points (4 core files) │ ├── main.cpp # Primary application entry, thread initialization │ ├── SQLite3_Server.cpp # Embedded configuration database diff --git a/doc/architecture/RELEASE-PIPELINE.md b/doc/architecture/RELEASE-PIPELINE.md new file mode 100644 index 000000000..4e1c8a11c --- /dev/null +++ b/doc/architecture/RELEASE-PIPELINE.md @@ -0,0 +1,696 @@ +# ProxySQL Release Pipeline Documentation + +## Executive Summary + +ProxySQL employs a sophisticated multi-tier release pipeline that automates package building, testing, and distribution across multiple Linux distributions and architectures. The pipeline integrates GitHub Actions, Jenkins automation *(Internal System)*, Docker-based builds, and comprehensive quality gates to ensure reliable releases for both development snapshots and production versions. + +## Release Pipeline Architecture + +```mermaid +graph TB + subgraph "Development Flow" + DEV[Development
Feature Branches] + PR[Pull Request
Review & Test] + MERGE[Merge to
Version Branch] + end + + subgraph "Build Triggers" + PUSH[Git Push
Version Branch] + TAG[Git Tag
Release Version] + MANUAL[Manual
Workflow Dispatch] + JENKINS[Jenkins Build
Internal System] + end + + subgraph "Build Pipeline" + DETECT[Version Detection
git describe] + MATRIX[Build Matrix
Multi-Platform] + DOCKER[Docker Builds
Containerized] + PACKAGE[Package Creation
RPM/DEB] + end + + subgraph "Testing & Validation" + PKGTEST[Package Tests
Installation] + INTTEST[Integration Tests
3rd Party] + SMOKE[Smoke Tests
Basic Function] + SIGN[Package Signing
SHA1 Hash] + end + + subgraph "Distribution" + GHREL[GitHub Releases
Public Downloads] + REPO[Repository Storage
Internal System] + DOCKER_HUB[Docker Hub
Container Images] + NOTIFY[Notifications
Release Notes] + end + + DEV --> PR + PR --> MERGE + MERGE --> PUSH + + PUSH --> DETECT + TAG --> DETECT + MANUAL --> DETECT + JENKINS --> DETECT + + DETECT --> MATRIX + MATRIX --> DOCKER + DOCKER --> PACKAGE + + PACKAGE --> PKGTEST + PKGTEST --> INTTEST + INTTEST --> SMOKE + SMOKE --> SIGN + + SIGN --> GHREL + SIGN --> REPO + SIGN --> DOCKER_HUB + GHREL --> NOTIFY + + style DETECT fill:#f9f,stroke:#333,stroke-width:4px + style PACKAGE fill:#bbf,stroke:#333,stroke-width:2px + style GHREL fill:#bfb,stroke:#333,stroke-width:2px +``` + +## Version Management Strategy + +### Version Numbering Scheme + +**Format**: `MAJOR.MINOR.PATCH-COMMITS-gHASH` + +```bash +# Version detection via git describe +GIT_VERSION=$(git describe --long --abbrev=7) +# Example: 2.7.0-404-g6000ede + +# Components: +# 2.7.0 - Last tagged version +# 404 - Commits since tag +# g6000ede - Git commit hash (g prefix) +``` + +### Branch Strategy + +| Branch Type | Purpose | Example | Release Type | +|-------------|---------|---------|--------------| +| **Main Development** | Active development | `v3.0` | Development snapshots | +| **Stable** | Production releases | `v2.7` | Official releases | +| **Maintenance** | Bug fixes | `2.x` | Patch releases | +| **Feature** | New features | `feature/pgsql-sasl` | Not released | + +### Tagging Convention + +```bash +# Production release tags +git tag -a v2.7.0 -m "Release version 2.7.0" +git push origin v2.7.0 + +# Release candidate tags +git tag -a v2.7.0-rc1 -m "Release candidate 1 for 2.7.0" + +# Development snapshot tags (automated) +v3.0-head # Latest development build +``` + +## Build Infrastructure + +### GitHub Actions Package Build + +**Workflow**: `CI-package-build.yml` + +```yaml +name: package-build +on: + workflow_dispatch: + workflow_run: + workflows: ["trigger"] + types: [completed] + +jobs: + get-info: + outputs: + is_tag: ${{ steps.tags.outputs.is_tag }} + is_github_branch: ${{ steps.branches.outputs.is_github_branch }} + + amd64-packages: + strategy: + matrix: + dist: [centos9, fedora40, fedora41, debian12, ubuntu22, ubuntu24, + opensuse15, almalinux8, almalinux9] + build: ["", "clang", "dbg"] +``` + +### Jenkins Build System *(Internal System)* + +**Location**: `priv-infra/jenkins-build-scripts/` + +**Main Build Scripts**: +```bash +# Package build orchestrator (Internal) +package-build.bash +├── Version detection +├── Docker container setup +├── Multi-architecture builds +├── Package transfer +└── Repository management + +# Build script (Internal) +build.bash +├── Dependency compilation +├── ProxySQL compilation +├── Test execution +├── Coverage collection +└── Artifact packaging +``` + +### Docker Build Environments + +**Build Container Matrix**: + +```mermaid +graph LR + subgraph "AMD64 Builds" + C9[CentOS 9] + F40[Fedora 40] + F41[Fedora 41] + D12[Debian 12] + U22[Ubuntu 22] + U24[Ubuntu 24] + OS15[OpenSUSE 15] + AL8[AlmaLinux 8] + AL9[AlmaLinux 9] + end + + subgraph "ARM64 Builds" + C9A[CentOS 9 ARM] + D12A[Debian 12 ARM] + U22A[Ubuntu 22 ARM] + U24A[Ubuntu 24 ARM] + end + + subgraph "Build Variants" + STD[Standard Build
ClickHouse] + CLG[Clang Build
Clang Compiler] + DBG[Debug Build
Debug Symbols] + end + + C9 --> STD + C9 --> CLG + C9 --> DBG + + C9A --> STD + C9A --> DBG +``` + +**Docker Image Specifications**: +```dockerfile +# Build image example: proxysql/packaging:build-debian12-v3.0 +FROM debian:12 +RUN apt-get update && apt-get install -y \ + build-essential cmake git \ + libssl-dev libmysqlclient-dev \ + libpq-dev python3 python3-pip +``` + +## Package Building Process + +### Build Execution Flow + +```mermaid +sequenceDiagram + participant GIT as Git Repository + participant CI as GitHub Actions + participant DOCKER as Docker Build + participant BUILD as Build Process + participant TEST as Package Test + participant DIST as Distribution + + GIT->>CI: Push/Tag Event + CI->>CI: Detect Version + CI->>DOCKER: Start Container + DOCKER->>BUILD: Execute Build + + BUILD->>BUILD: Compile Dependencies + BUILD->>BUILD: Build ProxySQL + BUILD->>BUILD: Create Package + + BUILD->>TEST: Install Package + TEST->>TEST: Validate Function + TEST->>TEST: Run Smoke Tests + + TEST->>DIST: Upload Package + DIST->>DIST: Generate SHA1 + DIST->>DIST: Publish Release +``` + +### Package Creation Details + +**RPM Package Building**: +```bash +# RPM spec generation +Version: ${CURVER} +Release: 1 +BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root + +# Package creation +rpmbuild -bb proxysql.spec +# Output: proxysql-2.7.0-1.x86_64.rpm +``` + +**DEB Package Building**: +```bash +# Debian control file +Package: proxysql +Version: ${CURVER}-${GIT_VERSION} +Architecture: amd64 + +# Package creation +dpkg-deb --build proxysql/ +# Output: proxysql_2.7.0-1_amd64.deb +``` + +### Build Optimization + +**NOOP Detection**: +```bash +# Skip unchanged builds (Internal) +if [[ "$BUILD_TYPE" == "noop" ]]; then + echo "No changes detected, skipping build" + exit 0 +fi +``` + +**Parallel Building**: +```makefile +MAKEFLAGS = -j$(shell nproc) +PARALLEL_JOBS = $(shell nproc) +``` + +## Quality Gates and Testing + +### Package Testing Framework + +**Location**: `priv-infra/proxysql-package-tests/` *(Internal System)* + +```bash +# Package test execution (Internal) +package-tester.bash +├── Docker environment setup +├── Package installation test +├── Service startup validation +├── Basic functionality check +├── Configuration test +└── Uninstallation verification +``` + +### Quality Gate Stages + +| Stage | Validation | Success Criteria | Failure Action | +|-------|------------|------------------|----------------| +| **Build** | Compilation success | Zero errors | Abort pipeline | +| **Package** | Package creation | Valid RPM/DEB | Abort pipeline | +| **Install** | Installation test | Clean install | Abort pipeline | +| **Startup** | Service startup | ProxySQL running | Abort pipeline | +| **Function** | Basic operations | Query execution | Abort pipeline | +| **Integration** | 3rd party tests | Connector tests pass | Warning only | +| **Performance** | Benchmark tests | No regression | Warning only | + +### Testing Matrix + +```mermaid +graph TB + subgraph "Package Tests" + INST[Installation
Package Manager] + START[Service Start
systemctl/init.d] + CONN[Connection
MySQL Protocol] + ADMIN[Admin Interface
Port 6032] + UNINST[Uninstall
Clean Removal] + end + + subgraph "Per Distribution" + RPM[RPM-based
CentOS, Fedora] + DEB[DEB-based
Debian, Ubuntu] + SUSE[OpenSUSE
zypper] + end + + INST --> RPM + INST --> DEB + INST --> SUSE + + START --> RPM + START --> DEB + START --> SUSE +``` + +## Artifact Management + +### Package Naming Convention + +**RPM Packages**: +``` +proxysql-{VERSION}-1.{DISTRO}.{ARCH}.rpm +proxysql-2.7.0-1.centos9.x86_64.rpm +proxysql-2.7.0-1.centos9.aarch64.rpm +proxysql-debuginfo-2.7.0-1.centos9.x86_64.rpm +``` + +**DEB Packages**: +``` +proxysql_{VERSION}-{RELEASE}_{ARCH}.deb +proxysql_2.7.0-ubuntu22_amd64.deb +proxysql_2.7.0-ubuntu22_arm64.deb +proxysql-dbg_2.7.0-ubuntu22_amd64.deb +``` + +### SHA1 Hash Generation + +```bash +# Extract binary and generate hash +rpm2cpio package.rpm | cpio -idmv +sha1sum usr/bin/proxysql > package.rpm.id-hash + +# For DEB packages +ar x package.deb +tar -xf data.tar.gz +sha1sum usr/bin/proxysql > package.deb.id-hash +``` + +### Artifact Storage + +**GitHub Releases**: +```bash +# Development snapshots +Release: v3.0-head +Assets: + - proxysql-3.0.0-dev-centos9.x86_64.rpm + - proxysql_3.0.0-dev-ubuntu22_amd64.deb + - SHA256SUMS + +# Production releases +Release: v2.7.0 +Assets: + - Source code (zip) + - Source code (tar.gz) + - All distribution packages + - Release notes +``` + +**Repository Storage** *(Internal System)*: +``` +/data/repos/ProxySQL-head/ +├── binaries-{VERSION}/ +│ ├── centos9/ +│ ├── debian12/ +│ ├── ubuntu22/ +│ └── ... +└── archive/ + └── old-versions/ +``` + +## Docker Image Release + +### Container Build Pipeline + +```dockerfile +# Production Dockerfile +FROM alpine:latest +COPY proxysql /usr/bin/ +EXPOSE 6033 6032 6080 +ENTRYPOINT ["proxysql", "-f", "-D", "/var/lib/proxysql"] +``` + +### Docker Hub Publishing + +```bash +# Build and tag images +docker build -t proxysql/proxysql:latest . +docker tag proxysql/proxysql:latest proxysql/proxysql:2.7.0 +docker tag proxysql/proxysql:latest proxysql/proxysql:2.7 + +# Multi-architecture manifest +docker manifest create proxysql/proxysql:2.7.0 \ + proxysql/proxysql:2.7.0-amd64 \ + proxysql/proxysql:2.7.0-arm64 + +# Push to Docker Hub +docker push proxysql/proxysql:2.7.0 +docker manifest push proxysql/proxysql:2.7.0 +``` + +## Release Automation + +### CI/CD Trigger Configuration + +```yaml +# Automatic release on tag +on: + push: + tags: + - 'v*.*.*' + branches: + - 'v[0-9].[0-9x]+.?[0-9xy]?[0-9]?' + +# Manual release trigger +on: + workflow_dispatch: + inputs: + version: + description: 'Version to release' + required: true + release_type: + description: 'Release type' + type: choice + options: + - production + - candidate + - snapshot +``` + +### Release Workflow Stages + +```mermaid +graph LR + subgraph "Pre-Release" + FREEZE[Code Freeze] + TEST[Final Testing] + APPROVE[Release Approval] + end + + subgraph "Release" + TAG[Create Tag] + BUILD[Build Packages] + PUBLISH[Publish Assets] + end + + subgraph "Post-Release" + ANNOUNCE[Announcement] + DOCS[Update Docs] + MONITOR[Monitor Issues] + end + + FREEZE --> TEST + TEST --> APPROVE + APPROVE --> TAG + TAG --> BUILD + BUILD --> PUBLISH + PUBLISH --> ANNOUNCE + ANNOUNCE --> DOCS + DOCS --> MONITOR +``` + +## Release Notes Generation + +### Automated Changelog + +```bash +# Generate changelog from commits +git log v2.6.0..v2.7.0 --pretty=format:"* %s (%an)" \ + --grep="^feat\|^fix\|^perf" > CHANGELOG.md + +# Categories: +# feat: New features +# fix: Bug fixes +# perf: Performance improvements +# docs: Documentation updates +# test: Test additions +``` + +### Release Note Template + +```markdown +# ProxySQL v2.7.0 Release Notes + +## Highlights +- Major feature additions +- Performance improvements +- Security enhancements + +## New Features +- Feature 1: Description +- Feature 2: Description + +## Bug Fixes +- Fix 1: Issue #XXX - Description +- Fix 2: Issue #YYY - Description + +## Breaking Changes +- Change 1: Migration required + +## Deprecations +- Deprecated feature 1 + +## Contributors +- List of contributors + +## Downloads +- [CentOS/RHEL packages](link) +- [Debian/Ubuntu packages](link) +- [Docker images](link) +``` + +## Distribution Channels + +### Package Repositories + +**RPM Repositories** *(Future Enhancement)*: +```bash +# YUM repository configuration +[proxysql] +name=ProxySQL Repository +baseurl=https://repo.proxysql.com/centos/$releasever/ +gpgcheck=1 +gpgkey=https://repo.proxysql.com/RPM-GPG-KEY-proxysql +``` + +**APT Repositories** *(Future Enhancement)*: +```bash +# APT repository configuration +deb https://repo.proxysql.com/debian/ $(lsb_release -cs) main +``` + +### Direct Downloads + +- **GitHub Releases**: Primary distribution channel +- **ProxySQL Website**: Mirror of GitHub releases +- **Docker Hub**: Container images + +## Monitoring and Metrics + +### Release Metrics + +| Metric | Target | Measurement | Tool | +|--------|--------|-------------|------| +| Build Success Rate | >95% | Successful builds/Total | GitHub Actions | +| Package Test Pass Rate | 100% | Passed tests/Total | Jenkins *(Internal)* | +| Release Cycle Time | <2 hours | Tag to publish | CI/CD Pipeline | +| Download Count | Tracked | Downloads per release | GitHub API | +| Issue Report Rate | <5/release | Issues per release | GitHub Issues | + +### Release Health Dashboard + +```mermaid +graph TB + subgraph "Release Health Indicators" + DOWNLOAD[Download Metrics
GitHub/Docker] + ISSUES[Issue Reports
Bug Tracking] + CRASH[Crash Reports
Core Dumps] + PERF[Performance
Benchmarks] + end + + subgraph "Actions" + HOTFIX[Hotfix Release] + PATCH[Patch Release] + ROLLBACK[Version Rollback] + end + + ISSUES --> HOTFIX + CRASH --> HOTFIX + PERF --> PATCH + + style DOWNLOAD fill:#bfb + style ISSUES fill:#ffb + style CRASH fill:#fbb +``` + +## Security Considerations + +### Package Signing *(Future Enhancement)* + +```bash +# GPG signing for RPM +rpmsign --addsign proxysql-2.7.0-1.x86_64.rpm + +# GPG signing for DEB +dpkg-sig --sign builder proxysql_2.7.0-1_amd64.deb + +# Checksum generation +sha256sum proxysql-* > SHA256SUMS +gpg --armor --detach-sign SHA256SUMS +``` + +### Vulnerability Scanning + +- **Container scanning**: Trivy, Snyk integration +- **Dependency scanning**: GitHub Dependabot +- **Binary scanning**: Static analysis tools + +## Rollback Procedures + +### Version Rollback Strategy + +```bash +# Package rollback (RPM) +yum downgrade proxysql-2.6.0 + +# Package rollback (DEB) +apt-get install proxysql=2.6.0 + +# Docker rollback +docker pull proxysql/proxysql:2.6.0 +docker stop proxysql && docker rm proxysql +docker run -d --name proxysql proxysql/proxysql:2.6.0 +``` + +### Emergency Response + +1. **Issue Detection**: Monitor crash reports and issues +2. **Impact Assessment**: Evaluate severity and scope +3. **Rollback Decision**: Determine if rollback needed +4. **Communication**: Notify users of issues +5. **Hotfix Release**: Expedited patch release + +## Best Practices + +### Release Checklist + +- [ ] All tests passing in CI/CD +- [ ] Version number updated +- [ ] Changelog updated +- [ ] Documentation updated +- [ ] Release notes prepared +- [ ] Package tests completed +- [ ] Security scan completed +- [ ] Performance benchmarks verified +- [ ] Rollback plan documented +- [ ] Communication plan ready + +### Release Cadence + +| Release Type | Frequency | Lead Time | Testing Period | +|--------------|-----------|-----------|----------------| +| Major (x.0.0) | 6-12 months | 4 weeks | 2 weeks | +| Minor (x.y.0) | 2-3 months | 2 weeks | 1 week | +| Patch (x.y.z) | As needed | 1 week | 3 days | +| Hotfix | Emergency | Immediate | Minimal | + +## Future Enhancements + +1. **GPG Signing**: Implement package signing for security +2. **Repository Hosting**: Official APT/YUM repositories +3. **Automated Rollback**: Automatic rollback on critical issues +4. **A/B Testing**: Gradual rollout with metrics +5. **Release Analytics**: Detailed usage and adoption metrics +6. **CD Pipeline**: Continuous deployment for development builds +7. **Multi-Cloud Distribution**: CDN for global distribution + +--- + +*This document represents the current state of ProxySQL's release pipeline. For the latest updates, refer to the GitHub Actions workflows and Jenkins build scripts *(Internal System)* in the repository.* \ No newline at end of file diff --git a/doc/architecture/VISUAL-GUIDE.md b/doc/architecture/VISUAL-GUIDE.md index 8b626d1db..13e83173f 100644 --- a/doc/architecture/VISUAL-GUIDE.md +++ b/doc/architecture/VISUAL-GUIDE.md @@ -561,6 +561,91 @@ graph TB style RETURN fill:#e8f5e8 ``` +## Advanced Authentication Flow + +### Multi-Stage Authentication State Machine + +```mermaid +stateDiagram-v2 + [*] --> Handshake: Client Connect + + Handshake --> SSLNegotiation: SSL Required + Handshake --> PluginSelect: No SSL + + SSLNegotiation --> PluginSelect: SSL Established + + PluginSelect --> NativeAuth: mysql_native_password + PluginSelect --> CachingSHA2: caching_sha2_password + PluginSelect --> ClearPass: mysql_clear_password + PluginSelect --> SCRAM: PostgreSQL SCRAM + + NativeAuth --> CheckCache: SHA1 Hash + CheckCache --> CacheHit: Found + CheckCache --> Backend: Not Found + + CachingSHA2 --> FastAuth: Cached + CachingSHA2 --> FullAuth: Not Cached + + ClearPass --> LDAP: LDAP Integration + SCRAM --> SASLExchange: SASL Protocol + + CacheHit --> Authenticated + Backend --> UpdateCache: Success + UpdateCache --> Authenticated + FastAuth --> Authenticated + FullAuth --> Authenticated + LDAP --> Authenticated + SASLExchange --> Authenticated + + Authenticated --> SessionEstablished + SessionEstablished --> [*] +``` + +### Query Digest and Rule Processing + +```mermaid +graph TB + subgraph "Query Digest Pipeline" + QUERY[SQL Query] --> SIZE{Size > 100KB?} + SIZE -->|Yes| FAST_PATH[Fast Digest
Optimized Path] + SIZE -->|No| NORMAL[Normal Digest] + + FAST_PATH --> THREAD_POOL[4 Digest Threads] + NORMAL --> COMPUTE[Compute Hash] + THREAD_POOL --> DIGEST_RESULT[Digest Value] + COMPUTE --> DIGEST_RESULT + end + + subgraph "Rule Processing Engine" + DIGEST_RESULT --> LOAD_RULES[Load Query Rules] + LOAD_RULES --> RULE_LOOP{For Each Rule} + + RULE_LOOP --> MATCH_USER[Check Username] + MATCH_USER --> MATCH_SCHEMA[Check Schema] + MATCH_SCHEMA --> MATCH_ADDR[Check Client Addr] + MATCH_ADDR --> MATCH_PATTERN[Check Pattern] + + MATCH_PATTERN -->|Match| APPLY_ACTIONS[Apply Actions] + MATCH_PATTERN -->|No Match| NEXT_RULE[Next Rule] + + APPLY_ACTIONS --> WEIGHT{Has Weight?} + WEIGHT -->|Yes| WEIGHTED_ROUTE[Weighted Distribution] + WEIGHT -->|No| DIRECT_ROUTE[Direct Route] + + APPLY_ACTIONS --> MIRROR{Mirror?} + MIRROR -->|Yes| MIRROR_SETUP[Setup Mirror HG] + + APPLY_ACTIONS --> CHAIN{Chain Rules?} + CHAIN -->|Yes| SET_FLAG[Set next_query_flagIN] + + NEXT_RULE --> RULE_LOOP + end + + style FAST_PATH fill:#e8f5e8 + style THREAD_POOL fill:#fff3e0 + style WEIGHTED_ROUTE fill:#e1f5fe +``` + ## Protocol Sequence Diagrams ### MySQL Connection and Query Sequence @@ -780,6 +865,93 @@ graph LR ## Connection Pooling Architecture +### Advanced Pool Management + +```mermaid +graph TB + subgraph "Connection Pool Internals" + subgraph "Pool Selection Criteria" + REQ[Connection Request] --> CRITERIA{Selection Criteria} + CRITERIA --> LAG[Max Lag Check] + CRITERIA --> GTID[GTID Position] + CRITERIA --> LATENCY[Latency Score] + CRITERIA --> WARM[Connection Warming] + end + + subgraph "Connection States" + FREE[Free Pool
ConnectionsFree] + USED[Used Pool
ConnectionsUsed] + WARMING[Warming Pool
Pre-established] + EXPIRED[Expired
To be closed] + end + + subgraph "Pool Algorithms" + ALG1[get_MyConn_from_pool()] + ALG1 --> CHECK_LAG{Lag < max_lag_ms?} + CHECK_LAG -->|Yes| CHECK_GTID{GTID Match?} + CHECK_LAG -->|No| REJECT1[Reject Connection] + CHECK_GTID -->|Yes| CHECK_LATENCY{Latency OK?} + CHECK_GTID -->|No| FIND_NEXT[Find Next] + CHECK_LATENCY -->|Yes| ASSIGN[Assign Connection] + CHECK_LATENCY -->|No| CREATE_NEW[Create New] + end + + LAG --> FREE + GTID --> FREE + LATENCY --> FREE + WARM --> WARMING + + FREE --> USED + USED --> FREE + WARMING --> FREE + FREE --> EXPIRED + USED --> EXPIRED + end + + style REQ fill:#e1f5fe + style FREE fill:#e8f5e8 + style USED fill:#ffebee + style ASSIGN fill:#c8e6c9 +``` + +### Pool Statistics and Metrics + +```mermaid +graph LR + subgraph "Connection Pool Metrics" + subgraph "Prometheus Counters" + PC1[connection_pool_conn_err] + PC2[connection_pool_conn_ok] + PC3[connection_pool_queries] + end + + subgraph "Prometheus Gauges" + PG1[connection_pool_conn_free] + PG2[connection_pool_conn_used] + PG3[connection_pool_latency_us] + end + + subgraph "Statistics Tables" + ST1[stats_mysql_connection_pool] + ST2[Connection Efficiency] + ST3[Per-Backend Metrics] + end + end + + PC1 --> ST1 + PC2 --> ST1 + PC3 --> ST2 + PG1 --> ST3 + PG2 --> ST3 + PG3 --> ST3 + + style PC1 fill:#ffebee + style PG1 fill:#e8f5e8 + style ST1 fill:#fff3e0 +``` + +## Connection Pooling Architecture + ### Pool Management Strategy ```mermaid @@ -1169,6 +1341,178 @@ graph TB style SECRET fill:#ffebee ``` +## Monitor Module Architecture + +### Work Queue Pattern + +```mermaid +graph TB + subgraph "Monitor Thread Pool" + subgraph "Work Queue System" + QUEUE[wqueue] + WT1[Worker Thread 1] + WT2[Worker Thread 2] + WTN[Worker Thread N] + end + + subgraph "Monitor Tasks" + CONNECT[Connect Check] + PING[Ping Check] + READONLY[Read-Only Check] + REPLAG[Replication Lag] + GALERA[Galera Status] + GROUP_REP[Group Replication] + end + + subgraph "Task Processing" + CONNECT --> QUEUE + PING --> QUEUE + READONLY --> QUEUE + REPLAG --> QUEUE + GALERA --> QUEUE + GROUP_REP --> QUEUE + + QUEUE --> WT1 + QUEUE --> WT2 + QUEUE --> WTN + end + end + + subgraph "Results Processing" + WT1 --> UPDATE[Update Server Status] + WT2 --> UPDATE + WTN --> UPDATE + + UPDATE --> SHUN{Shun Server?} + SHUN -->|Yes| SET_SHUNNED[Mark SHUNNED] + SHUN -->|No| SET_ONLINE[Keep ONLINE] + end + + style QUEUE fill:#e1f5fe + style UPDATE fill:#e8f5e8 +``` + +### Health Check State Machine + +```mermaid +stateDiagram-v2 + [*] --> ONLINE: Initial State + + ONLINE --> CHECKING: Monitor Interval + + CHECKING --> CONNECT_TEST: Perform Check + CONNECT_TEST --> PING_TEST: Connect OK + CONNECT_TEST --> SHUNNED: Connect Fail + + PING_TEST --> LAG_CHECK: Ping OK + PING_TEST --> SHUNNED: Ping Fail + + LAG_CHECK --> ONLINE: Lag < Threshold + LAG_CHECK --> SHUNNED_LAG: Lag > Threshold + + SHUNNED --> RECOVERY: Retry Interval + SHUNNED_LAG --> RECOVERY: Retry Interval + + RECOVERY --> CONNECT_TEST: Retry Check + + SHUNNED_LAG --> ONLINE: Lag Resolved + SHUNNED --> ONLINE: Connection Restored + + ONLINE --> OFFLINE_SOFT: Admin Command + OFFLINE_SOFT --> ONLINE: Admin Command + + ONLINE --> OFFLINE_HARD: Admin Command + OFFLINE_HARD --> [*]: Removed +``` + +## Cluster Synchronization Architecture + +### Checksum-Based Sync Mechanism + +```mermaid +graph TB + subgraph "Node A" + A_CONFIG[Configuration] + A_CHECKSUM[Calculate Checksum] + A_EPOCH[Epoch: 100] + A_VERSION[Version: 2] + + A_CONFIG --> A_CHECKSUM + A_CHECKSUM --> A_CS[Checksum: 0xABCD] + end + + subgraph "Node B" + B_CONFIG[Configuration] + B_CHECKSUM[Calculate Checksum] + B_EPOCH[Epoch: 101] + B_VERSION[Version: 2] + + B_CONFIG --> B_CHECKSUM + B_CHECKSUM --> B_CS[Checksum: 0xEF01] + end + + subgraph "Sync Decision" + COMPARE{Compare Checksums} + A_CS --> COMPARE + B_CS --> COMPARE + + COMPARE -->|Different| CHECK_EPOCH{Check Epoch} + COMPARE -->|Same| NO_SYNC[No Sync Needed] + + CHECK_EPOCH -->|B > A| SYNC_FROM_B[A syncs from B] + CHECK_EPOCH -->|A > B| SYNC_FROM_A[B syncs from A] + + SYNC_FROM_B --> APPLY_B[Apply B's Config to A] + SYNC_FROM_A --> APPLY_A[Apply A's Config to B] + end + + style A_EPOCH fill:#e1f5fe + style B_EPOCH fill:#e8f5e8 + style SYNC_FROM_B fill:#fff3e0 +``` + +### Cluster Module Synchronization + +```mermaid +graph LR + subgraph "Configuration Modules" + M1[mysql_servers] + M2[mysql_users] + M3[mysql_query_rules] + M4[mysql_variables] + M5[proxysql_servers] + end + + subgraph "Checksum Calculation" + M1 --> CS1[Checksum 1] + M2 --> CS2[Checksum 2] + M3 --> CS3[Checksum 3] + M4 --> CS4[Checksum 4] + M5 --> CS5[Checksum 5] + end + + subgraph "Sync Decision per Module" + CS1 --> DIFF1{Diff Count} + CS2 --> DIFF2{Diff Count} + CS3 --> DIFF3{Diff Count} + + DIFF1 -->|>= threshold| SYNC1[Sync Module 1] + DIFF2 -->|>= threshold| SYNC2[Sync Module 2] + DIFF3 -->|>= threshold| SYNC3[Sync Module 3] + end + + subgraph "Global Checksum" + CS1 --> GLOBAL[Global Checksum] + CS2 --> GLOBAL + CS3 --> GLOBAL + CS4 --> GLOBAL + CS5 --> GLOBAL + end + + style M1 fill:#e1f5fe + style GLOBAL fill:#fff3e0 +``` + ## Performance Optimization Points ### Critical Path Optimizations