pull/5115/head
noizu 8 months ago
parent 5f15d971fe
commit 89849d0dc0

@ -102,6 +102,47 @@ Claude Code provides specialized agents to help with different aspects of ProxyS
"Use gpt-qa to create comprehensive tests for PgSQL prepared statements"
```
### Running Agents in Parallel for Faster Analysis
For optimal performance when analyzing ProxySQL's codebase, run multiple agents simultaneously:
```bash
# RECOMMENDED: Parallel agent execution for comprehensive analysis
"Run gopher-scout and system-digest in parallel to analyze the MySQL session handling architecture"
# Parallel exploration of different code areas
"Deploy gopher-scout in parallel to:
1. Find all MySQL authentication methods in lib/MySQL_Authentication.cpp
2. Analyze connection pooling in lib/MySQL_HostGroups_Manager.cpp
3. Trace query routing logic in lib/MySQL_Query_Processor.cpp"
# Parallel architecture + implementation analysis
"Run these agents in parallel:
- system-digest to analyze the overall threading architecture
- gopher-scout to find specific thread pool implementations in MySQL_Thread.cpp
- gopher-scout to examine PostgreSQL thread handling in PgSQL_Thread.cpp"
# Parallel test creation and feature implementation
"Execute in parallel:
- tdd-driven-builder to implement the new feature
- gpt-qa to create comprehensive test coverage
- gopher-scout to find similar existing implementations for reference"
```
#### Performance Benefits of Parallel Execution
1. **Faster Results**: Agents work concurrently, reducing total analysis time by 3-5x
2. **Comprehensive Coverage**: Different agents can explore different aspects simultaneously
3. **Cross-Reference**: Results from multiple agents provide richer context
4. **Efficient Resource Use**: Claude Code optimizes parallel execution automatically
#### Best Practices for Parallel Agent Use
- **Combine complementary agents**: Pair high-level (system-digest) with detailed (gopher-scout) analysis
- **Partition search spaces**: Have different gopher-scout instances scan different directories
- **Mix analysis types**: Run architecture analysis alongside implementation searches
- **Batch related queries**: Group related exploration tasks for parallel execution
## Common Development Workflows
### Understanding Code Flow

@ -2,28 +2,28 @@
## Executive Summary
ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/17. It implements a multi-threaded architecture with connection pooling, query routing, caching, and monitoring capabilities.
ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/17. It implements a multi-threaded architecture with connection pooling, query routing, caching, and monitoring.
## System Architecture
### Core Design Patterns
1. **Multi-Threaded Worker Model**
- MySQL worker threads (`MySQL_Thread`) handling client connections
- MySQL worker threads (`MySQL_Thread`) handle client connections
- PgSQL worker threads (`PgSQL_Thread`) for PostgreSQL support
- Admin threads for configuration management
- Monitor threads for backend health checking
- Idle connection management threads (when `IDLE_THREADS` enabled)
2. **Event-Driven I/O**
- Uses `libev` for efficient event loop management
- Poll-based multiplexing for handling multiple connections per thread
- Uses `libev` for event loop management
- Poll-based multiplexing handles multiple connections per thread
- Epoll support for idle thread management on Linux
3. **Connection Pooling & Multiplexing**
- Per-hostgroup connection pools
- Connection multiplexing to reduce backend connections
- Smart connection reuse based on session state
- Connection reuse based on session state
4. **Protocol Implementation**
- Full MySQL protocol implementation (`MySQL_Protocol`)
@ -36,9 +36,9 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/
- **File**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/main.cpp`
- **Responsibilities**:
- Process initialization and daemonization
- Loading configuration from `proxysql.cfg`
- Creating global variables structure
- Starting all subsystems
- Loads configuration from `proxysql.cfg`
- Creates global variables structure
- Starts all subsystems
### 2. Thread Architecture
@ -57,7 +57,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/
#### MySQL Threads (`MySQL_Thread`)
- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Thread.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Thread.h`
- **Key Features**:
- Worker threads handling MySQL client connections
- Worker threads handle MySQL client connections
- Session management and query processing
- Connection pool interaction
- Thread-local statistics for lock-free updates
@ -94,7 +94,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/
#### MySQL HostGroups Manager
- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_HostGroups_Manager.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_HostGroups_Manager.h`
- **Key Concepts**:
- Hostgroup: logical grouping of database servers
- Hostgroups logically group database servers
- Connection pool per hostgroup
- Server status tracking (ONLINE, SHUNNED, OFFLINE_SOFT, OFFLINE_HARD)
- Connection health monitoring
@ -109,7 +109,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
#### MySQL Query Processor
- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Query_Processor.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Query_Processor.h`
- **Features**:
- **Functions**:
- Rule-based query routing
- Query rewriting capabilities
- Query caching decisions
@ -148,7 +148,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
#### SQLite3 Server
- **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/SQLite3_Server.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/SQLite3_Server.h`
- **Purpose**: Provides SQL interface for admin operations
- **Purpose**: SQL interface for admin operations
#### Monitoring (`MySQL_Monitor`, `PgSQL_Monitor`)
- Backend health checking
@ -261,7 +261,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
- Latency-aware connection selection
- GTID-aware routing for consistency
2. **Advanced Pool Algorithms**:
2. **Pool Algorithms**:
```cpp
// Connection retrieval with multiple criteria
MySQL_Connection* get_MyConn_from_pool(
@ -273,11 +273,11 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
)
```
3. **Query Processing Optimizations**:
- **Fast Digest Path**: Optimized computation for queries > 100KB
- **Multi-threaded Digesting**: 4 dedicated threads for parallel processing
3. **Query Processing**:
- **Fast Digest Path**: Optimized for queries > 100KB
- **Multi-threaded Digesting**: 4 threads for parallel processing
- **Regex Caching**: Compiled patterns cached in `regex_engine1/2`
- **Digest Statistics**: Efficient tracking with minimal overhead
- **Digest Statistics**: Low-overhead tracking
4. **Memory Management**:
- **Buffer Pools**: Reusable buffers for packet handling
@ -285,7 +285,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
- **Result Buffering**: Configurable strategies
- **jemalloc Integration**: Optimized memory allocation
5. **Lock-Free Optimizations**:
5. **Lock-Free Structures**:
- Thread-local statistics counters
- Lock-free query digest maps
- Atomic operations for global counters
@ -325,8 +325,8 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
- **PostgreSQL SCRAM**: SASL/SCRAM-SHA-256 support
#### Authentication Caching
- SHA1 passwords cached in `GloMyAuth` for performance
- Clear text passwords cached for `caching_sha2_password` fast auth
- SHA1 passwords cached in `GloMyAuth`
- Passwords cached for `caching_sha2_password` fast authentication
- User attributes cached with JSON validation
- Per-user connection limits and routing rules
@ -340,7 +340,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD
## Query Processing Pipeline
### Query Digest System
- **Digest Computation**: Fast path for queries > 100KB
- **Digest Computation**: Optimized for queries > 100KB
- **Digest Structure**:
```cpp
struct QP_query_digest_stats {
@ -434,7 +434,7 @@ THEN sync_from_peer
## Architecture Extensions
The architecture supports:
Architecture supports:
- Additional database protocols
- Alternative caching strategies
- Custom routing algorithms

@ -2,13 +2,13 @@
## Executive Summary
ProxySQL is a database proxy server written in C++ that provides protocol-aware proxying for MySQL and PostgreSQL databases. The system implements a multi-threaded, event-driven architecture with connection pooling, query routing, and monitoring. Built with C++11/17, it uses a modular design with separation between protocol handlers, session management, and administrative interfaces, backed by an embedded SQLite3 configuration database.
ProxySQL is a database proxy server written in C++ providing protocol-aware proxying for MySQL and PostgreSQL databases. It implements a multi-threaded, event-driven architecture with connection pooling, query routing, and monitoring. Built with C++11/17, it uses modular design separating protocol handlers, session management, and administrative interfaces, backed by embedded SQLite3 configuration.
## Key Components
- **Dual-protocol architecture**: MySQL and PostgreSQL wire protocol implementations
- **Multi-threaded design**: Separate worker threads for connections, admin, monitoring, and clustering
- **Embedded configuration**: SQLite3-based three-tier configuration system (Disk → Memory → Runtime)
- **Dual protocols**: MySQL and PostgreSQL wire protocol implementations
- **Multi-threaded**: Separate worker threads for connections, admin, monitoring, and clustering
- **Configuration**: SQLite3 three-tier system (Disk → Memory → Runtime)
- **Performance features**: Lock-free structures, connection pooling, query caching
- **Enterprise features**: Clustering, monitoring, REST API, and Prometheus metrics
@ -113,28 +113,28 @@ graph LR
### Code Organization
```
https://github.com/sysown/proxysql/tree/v3.0.agentics/
├── src/ # Main entry points (4 core files)
│ ├── main.cpp # Primary application entry, thread initialization
│ ├── SQLite3_Server.cpp # Embedded configuration database
├── src/ # Main entry points (4 files)
│ ├── main.cpp # Application entry, thread initialization
│ ├── SQLite3_Server.cpp # Configuration database
│ ├── proxy_tls.cpp # TLS/SSL implementation
│ └── proxysql_global.cpp # Global variables and configuration
│ └── proxysql_global.cpp # Global variables, configuration
├── lib/ # Core library implementations (86 .cpp files)
│ ├── MySQL_*.cpp # MySQL protocol & management (20+ files)
│ │ ├── MySQL_Protocol.cpp # Wire protocol implementation
│ │ ├── MySQL_Session.cpp # Client session handling
│ │ ├── MySQL_Protocol.cpp # Wire protocol
│ │ ├── MySQL_Session.cpp # Session handling
│ │ ├── MySQL_HostGroups_Manager.cpp # Backend management
│ │ └── MySQL_Monitor.cpp # Health monitoring
│ │
│ ├── PgSQL_*.cpp # PostgreSQL protocol & management (15+ files)
│ │ ├── PgSQL_Protocol.cpp # Wire protocol implementation
│ │ ├── PgSQL_Session.cpp # Client session handling
│ │ └── PgSQL_Authentication.cpp # SASL/SCRAM support
│ │ ├── PgSQL_Protocol.cpp # Wire protocol v3
│ │ ├── PgSQL_Session.cpp # Session handling
│ │ └── PgSQL_Authentication.cpp # SASL/SCRAM
│ │
│ ├── ProxySQL_Admin*.cpp # Administrative interface (10+ files)
│ │ ├── ProxySQL_Admin.cpp # Main admin implementation
│ │ ├── ProxySQL_Admin_Stats.cpp # Statistics collection
│ │ └── ProxySQL_RESTAPI_Server.cpp # REST API endpoint
│ │ ├── ProxySQL_Admin.cpp # Admin implementation
│ │ ├── ProxySQL_Admin_Stats.cpp # Statistics
│ │ └── ProxySQL_RESTAPI_Server.cpp # REST API
│ │
│ ├── Base_*.cpp # Base infrastructure (5+ files)
│ ├── Query_*.cpp # Query processing & caching
@ -156,8 +156,8 @@ https://github.com/sysown/proxysql/tree/v3.0.agentics/
│ └── ... # 17 more dependencies
└── test/ # Test infrastructure
├── tap/tests/ # 220+ TAP test files
├── cluster/ # Cluster functionality tests
├── tap/tests/ # 220+ TAP tests
├── cluster/ # Cluster tests
└── PrepStmt/ # Prepared statement tests
```
@ -277,11 +277,11 @@ graph LR
└─────────────────┘
```
### Port Mapping and Interfaces
- **6033**: MySQL/PostgreSQL client connections (main proxy port)
- **6032**: Admin interface (MySQL-compatible protocol)
- **6080**: REST API endpoint (HTTP/JSON)
- **6070**: Web UI interface (optional)
### Port Mapping
- **6033**: MySQL/PostgreSQL client connections
- **6032**: Admin interface (MySQL protocol)
- **6080**: REST API (HTTP/JSON)
- **6070**: Web UI (optional)
### Docker Deployment Scenarios
```
@ -317,10 +317,10 @@ INSERT INTO mysql_query_rules (
```
### Health Monitoring
- **Automatic health checks**: Configurable intervals
- **Health checks**: Configurable intervals
- **Connection verification**: `monitor_ping_interval`
- **Replication lag detection**: For read/write splitting
- **Automatic shunning**: Remove unhealthy backends
- **Replication lag detection**: Read/write splitting
- **Automatic shunning**: Removes unhealthy backends
## Performance and Scaling Architecture
@ -359,10 +359,10 @@ graph TD
style BE fill:#e1f5fe
```
### Caching Architecture
- **Query Cache**: Configurable TTL, pattern-based
### Caching
- **Query Cache**: TTL and pattern-based
- **Prepared Statement Cache**: Metadata caching
- **Connection Attributes Cache**: Reduces authentication overhead
- **Connection Attributes Cache**: Reduced authentication overhead
## Testing and Quality Assurance
@ -411,11 +411,11 @@ mysql -h127.0.0.1 -P6032 -uadmin -padmin
## Architecture Features
1. **Protocol-Aware Proxy**: Understands and manipulates database protocols
2. **Runtime Reconfiguration**: Configuration changes without connection drops
3. **Load Balancing**: Weight-based, least-connections, round-robin algorithms
4. **Query Routing**: Regex-based rules, query rewriting, caching
5. **Integration**: Prometheus metrics, REST API, cluster support
6. **Security**: SSL/TLS, SQL injection detection, authentication plugins
7. **Performance**: Lock-free statistics, connection pooling, query cache
8. **High Availability**: Automatic failover, read/write splitting, replication lag detection
1. **Protocol-Aware**: Understands and manipulates database protocols
2. **Runtime Reconfiguration**: Changes without connection drops
3. **Load Balancing**: Weight-based, least-connections, round-robin
4. **Query Routing**: Regex rules, rewriting, caching
5. **Integration**: Prometheus, REST API, clustering
6. **Security**: SSL/TLS, SQL injection detection, authentication
7. **Performance**: Lock-free statistics, connection pooling, caching
8. **High Availability**: Failover, read/write splitting, lag detection

@ -104,42 +104,42 @@ graph TB
```
proxysql/
├── src/ [Main Entry Points - 4 files]
│ ├── main.cpp # Application entry, thread initialization
│ ├── SQLite3_Server.cpp # Embedded config database server
│ ├── main.cpp # Application entry, thread init
│ ├── SQLite3_Server.cpp # Config database server
│ ├── proxy_tls.cpp # TLS/SSL implementation
│ └── proxysql_global.cpp # Global variables and config
│ └── proxysql_global.cpp # Global variables, config
├── lib/ [Core Libraries - 86+ files]
│ ├── [MySQL Components - 25+ files]
│ │ ├── MySQL_Session.cpp # Client session management
│ │ ├── MySQL_Protocol.cpp # Wire protocol implementation
│ │ ├── MySQL_HostGroups_Manager.cpp # Backend server management
│ │ ├── MySQL_Session.cpp # Session management
│ │ ├── MySQL_Protocol.cpp # Wire protocol
│ │ ├── MySQL_HostGroups_Manager.cpp # Backend management
│ │ ├── MySQL_Monitor.cpp # Health monitoring
│ │ ├── MySQL_Authentication.cpp # Auth methods (native, sha256, etc)
│ │ ├── MySQL_Query_Processor.cpp # Query routing logic
│ │ ├── MySQL_Authentication.cpp # Auth methods
│ │ ├── MySQL_Query_Processor.cpp # Query routing
│ │ ├── MySQL_Query_Cache.cpp # Result caching
│ │ ├── MySQL_Thread.cpp # Thread pool management
│ │ ├── MySQL_Logger.cpp # Query/error logging
│ │ ├── MySQL_PreparedStatement.cpp # PS protocol handling
│ │ ├── MySQL_Thread.cpp # Thread pool
│ │ ├── MySQL_Logger.cpp # Logging
│ │ ├── MySQL_PreparedStatement.cpp # Prepared statements
│ │ └── MySQL_Variables.cpp # Session variables
│ │
│ ├── [PostgreSQL Components - 20+ files]
│ │ ├── PgSQL_Session.cpp # Client session management
│ │ ├── PgSQL_Session.cpp # Session management
│ │ ├── PgSQL_Protocol.cpp # Wire protocol v3
│ │ ├── PgSQL_HostGroups_Manager.cpp # Backend management
│ │ ├── PgSQL_Monitor.cpp # Health checks
│ │ ├── PgSQL_Authentication.cpp # SASL/SCRAM support
│ │ ├── PgSQL_Authentication.cpp # SASL/SCRAM
│ │ ├── PgSQL_Query_Processor.cpp # Query routing
│ │ ├── PgSQL_Query_Cache.cpp # Result caching
│ │ ├── PgSQL_Thread.cpp # Thread management
│ │ └── PgSQL_Logger.cpp # Logging
│ │
│ ├── [Base Infrastructure - 10+ files]
│ │ ├── Base_Session.cpp # Template base for sessions
│ │ ├── Base_Thread.cpp # Common threading
│ │ ├── Base_HostGroups_Manager.cpp # Template base for HG
│ │ ├── Query_Processor.cpp # Common query processing
│ │ └── Query_Cache.cpp # Common caching logic
│ │ ├── Base_Session.cpp # Session template base
│ │ ├── Base_Thread.cpp # Threading base
│ │ ├── Base_HostGroups_Manager.cpp # HostGroups base
│ │ ├── Query_Processor.cpp # Query processing base
│ │ └── Query_Cache.cpp # Caching base
│ │
│ ├── [Admin & Monitoring - 15+ files]
│ │ ├── ProxySQL_Admin.cpp # Admin interface (6032)
@ -150,25 +150,25 @@ proxysql/
│ │ └── ProxySQL_Config.cpp # Configuration management
│ │
│ └── [Supporting Libraries]
│ ├── Standard_Query_Cache.cpp # Query cache implementation
│ ├── MySQL_ResultSet.cpp # Result set handling
│ ├── Standard_Query_Cache.cpp # Query cache
│ ├── MySQL_ResultSet.cpp # Result sets
│ ├── network.cpp # Network utilities
│ ├── debug.cpp # Debug/logging utilities
│ └── libproxysql.a # Compiled static library (340MB+)
│ ├── debug.cpp # Debug utilities
│ └── libproxysql.a # Static library (340MB+)
├── include/ [Header Files - 89+ files]
│ ├── [Protocol Headers]
│ │ ├── MySQL_Protocol.h # MySQL protocol definitions
│ │ ├── PgSQL_Protocol.h # PostgreSQL protocol defs
│ │ ├── mysql_connection.h # MySQL connection handling
│ │ ├── MySQL_Protocol.h # MySQL protocol
│ │ ├── PgSQL_Protocol.h # PostgreSQL protocol
│ │ ├── mysql_connection.h # MySQL connections
│ │ └── pgsql_connection.h # PostgreSQL connections
│ │
│ ├── [Core Headers]
│ │ ├── proxysql.h # Main header
│ │ ├── proxysql_structs.h # Core data structures
│ │ ├── Base_Session.h # Session template base
│ │ ├── Base_Thread.h # Thread base class
│ │ └── Base_HostGroups_Manager.h # HG template base
│ │ ├── proxysql_structs.h # Data structures
│ │ ├── Base_Session.h # Session base
│ │ ├── Base_Thread.h # Thread base
│ │ └── Base_HostGroups_Manager.h # HostGroups base
│ │
│ └── [Utility Headers]
│ ├── btree_map.h # B-tree implementation
@ -191,13 +191,13 @@ proxysql/
│ └── [12 more libraries...]
└── test/ [Test Infrastructure]
├── tap/ # TAP test framework
│ └── tests/ # 220+ test files
│ ├── test_mysql_*.cpp # MySQL-specific tests
├── tap/ # TAP framework
│ └── tests/ # 220+ tests
│ ├── test_mysql_*.cpp # MySQL tests
│ ├── test_pgsql_*.cpp # PostgreSQL tests
│ └── test_admin_*.cpp # Admin interface tests
│ └── test_admin_*.cpp # Admin tests
├── cluster/ # Cluster tests
└── PrepStmt/ # Prepared statement tests
└── PrepStmt/ # Prepared statements
```
## Class Hierarchy Diagrams
@ -757,12 +757,12 @@ sequenceDiagram
## Thread Architecture
### Multi-Threaded Processing Model
### Thread Processing Model
```mermaid
graph TB
subgraph "Main Thread"
MAIN["main()"]
MAIN[main()]
INIT[Initialize]
CONFIG[Load Config]
START[Start Threads]
@ -865,7 +865,7 @@ graph LR
## Connection Pooling Architecture
### Advanced Pool Management
### Pool Management
```mermaid
graph TB
@ -1515,7 +1515,7 @@ graph LR
## Performance Optimization Points
### Critical Path Optimizations
### Performance Optimizations
```mermaid
graph LR

Loading…
Cancel
Save