From 89849d0dc02ab73287a3be2fc6f2240695ef1ade Mon Sep 17 00:00:00 2001 From: noizu Date: Thu, 11 Sep 2025 12:28:39 +0700 Subject: [PATCH] patch --- CLAUDE.md | 41 ++++++++++++ doc/architecture/ARCHITECTURE-OVERVIEW.md | 44 ++++++------- doc/architecture/PROJECT-LAYOUT.md | 74 +++++++++++----------- doc/architecture/VISUAL-GUIDE.md | 76 +++++++++++------------ 4 files changed, 138 insertions(+), 97 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 86e1aa57d..21e8c2361 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -102,6 +102,47 @@ Claude Code provides specialized agents to help with different aspects of ProxyS "Use gpt-qa to create comprehensive tests for PgSQL prepared statements" ``` +### Running Agents in Parallel for Faster Analysis + +For optimal performance when analyzing ProxySQL's codebase, run multiple agents simultaneously: + +```bash +# RECOMMENDED: Parallel agent execution for comprehensive analysis +"Run gopher-scout and system-digest in parallel to analyze the MySQL session handling architecture" + +# Parallel exploration of different code areas +"Deploy gopher-scout in parallel to: +1. Find all MySQL authentication methods in lib/MySQL_Authentication.cpp +2. Analyze connection pooling in lib/MySQL_HostGroups_Manager.cpp +3. Trace query routing logic in lib/MySQL_Query_Processor.cpp" + +# Parallel architecture + implementation analysis +"Run these agents in parallel: +- system-digest to analyze the overall threading architecture +- gopher-scout to find specific thread pool implementations in MySQL_Thread.cpp +- gopher-scout to examine PostgreSQL thread handling in PgSQL_Thread.cpp" + +# Parallel test creation and feature implementation +"Execute in parallel: +- tdd-driven-builder to implement the new feature +- gpt-qa to create comprehensive test coverage +- gopher-scout to find similar existing implementations for reference" +``` + +#### Performance Benefits of Parallel Execution + +1. **Faster Results**: Agents work concurrently, reducing total analysis time by 3-5x +2. **Comprehensive Coverage**: Different agents can explore different aspects simultaneously +3. **Cross-Reference**: Results from multiple agents provide richer context +4. **Efficient Resource Use**: Claude Code optimizes parallel execution automatically + +#### Best Practices for Parallel Agent Use + +- **Combine complementary agents**: Pair high-level (system-digest) with detailed (gopher-scout) analysis +- **Partition search spaces**: Have different gopher-scout instances scan different directories +- **Mix analysis types**: Run architecture analysis alongside implementation searches +- **Batch related queries**: Group related exploration tasks for parallel execution + ## Common Development Workflows ### Understanding Code Flow diff --git a/doc/architecture/ARCHITECTURE-OVERVIEW.md b/doc/architecture/ARCHITECTURE-OVERVIEW.md index 9e9a604bb..8d1d465a8 100644 --- a/doc/architecture/ARCHITECTURE-OVERVIEW.md +++ b/doc/architecture/ARCHITECTURE-OVERVIEW.md @@ -2,28 +2,28 @@ ## Executive Summary -ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/17. It implements a multi-threaded architecture with connection pooling, query routing, caching, and monitoring capabilities. +ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/17. It implements a multi-threaded architecture with connection pooling, query routing, caching, and monitoring. ## System Architecture ### Core Design Patterns 1. **Multi-Threaded Worker Model** - - MySQL worker threads (`MySQL_Thread`) handling client connections + - MySQL worker threads (`MySQL_Thread`) handle client connections - PgSQL worker threads (`PgSQL_Thread`) for PostgreSQL support - Admin threads for configuration management - Monitor threads for backend health checking - Idle connection management threads (when `IDLE_THREADS` enabled) 2. **Event-Driven I/O** - - Uses `libev` for efficient event loop management - - Poll-based multiplexing for handling multiple connections per thread + - Uses `libev` for event loop management + - Poll-based multiplexing handles multiple connections per thread - Epoll support for idle thread management on Linux 3. **Connection Pooling & Multiplexing** - Per-hostgroup connection pools - Connection multiplexing to reduce backend connections - - Smart connection reuse based on session state + - Connection reuse based on session state 4. **Protocol Implementation** - Full MySQL protocol implementation (`MySQL_Protocol`) @@ -36,9 +36,9 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ - **File**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/main.cpp` - **Responsibilities**: - Process initialization and daemonization - - Loading configuration from `proxysql.cfg` - - Creating global variables structure - - Starting all subsystems + - Loads configuration from `proxysql.cfg` + - Creates global variables structure + - Starts all subsystems ### 2. Thread Architecture @@ -57,7 +57,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ #### MySQL Threads (`MySQL_Thread`) - **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Thread.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Thread.h` - **Key Features**: - - Worker threads handling MySQL client connections + - Worker threads handle MySQL client connections - Session management and query processing - Connection pool interaction - Thread-local statistics for lock-free updates @@ -94,7 +94,7 @@ ProxySQL is a MySQL and PostgreSQL protocol-aware proxy server written in C++11/ #### MySQL HostGroups Manager - **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_HostGroups_Manager.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_HostGroups_Manager.h` - **Key Concepts**: - - Hostgroup: logical grouping of database servers + - Hostgroups logically group database servers - Connection pool per hostgroup - Server status tracking (ONLINE, SHUNNED, OFFLINE_SOFT, OFFLINE_HARD) - Connection health monitoring @@ -109,7 +109,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD #### MySQL Query Processor - **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/lib/MySQL_Query_Processor.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/MySQL_Query_Processor.h` -- **Features**: +- **Functions**: - Rule-based query routing - Query rewriting capabilities - Query caching decisions @@ -148,7 +148,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD #### SQLite3 Server - **Files**: `https://github.com/sysown/proxysql/tree/v3.0.agentics/src/SQLite3_Server.cpp`, `https://github.com/sysown/proxysql/tree/v3.0.agentics/include/SQLite3_Server.h` -- **Purpose**: Provides SQL interface for admin operations +- **Purpose**: SQL interface for admin operations #### Monitoring (`MySQL_Monitor`, `PgSQL_Monitor`) - Backend health checking @@ -261,7 +261,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD - Latency-aware connection selection - GTID-aware routing for consistency -2. **Advanced Pool Algorithms**: +2. **Pool Algorithms**: ```cpp // Connection retrieval with multiple criteria MySQL_Connection* get_MyConn_from_pool( @@ -273,11 +273,11 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ) ``` -3. **Query Processing Optimizations**: - - **Fast Digest Path**: Optimized computation for queries > 100KB - - **Multi-threaded Digesting**: 4 dedicated threads for parallel processing +3. **Query Processing**: + - **Fast Digest Path**: Optimized for queries > 100KB + - **Multi-threaded Digesting**: 4 threads for parallel processing - **Regex Caching**: Compiled patterns cached in `regex_engine1/2` - - **Digest Statistics**: Efficient tracking with minimal overhead + - **Digest Statistics**: Low-overhead tracking 4. **Memory Management**: - **Buffer Pools**: Reusable buffers for packet handling @@ -285,7 +285,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD - **Result Buffering**: Configurable strategies - **jemalloc Integration**: Optimized memory allocation -5. **Lock-Free Optimizations**: +5. **Lock-Free Structures**: - Thread-local statistics counters - Lock-free query digest maps - Atomic operations for global counters @@ -325,8 +325,8 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD - **PostgreSQL SCRAM**: SASL/SCRAM-SHA-256 support #### Authentication Caching -- SHA1 passwords cached in `GloMyAuth` for performance -- Clear text passwords cached for `caching_sha2_password` fast auth +- SHA1 passwords cached in `GloMyAuth` +- Passwords cached for `caching_sha2_password` fast authentication - User attributes cached with JSON validation - Per-user connection limits and routing rules @@ -340,7 +340,7 @@ ONLINE → SHUNNED (temporary failures) → OFFLINE_SOFT → OFFLINE_HARD ## Query Processing Pipeline ### Query Digest System -- **Digest Computation**: Fast path for queries > 100KB +- **Digest Computation**: Optimized for queries > 100KB - **Digest Structure**: ```cpp struct QP_query_digest_stats { @@ -434,7 +434,7 @@ THEN sync_from_peer ## Architecture Extensions -The architecture supports: +Architecture supports: - Additional database protocols - Alternative caching strategies - Custom routing algorithms diff --git a/doc/architecture/PROJECT-LAYOUT.md b/doc/architecture/PROJECT-LAYOUT.md index b8038d45c..39c05ae9d 100644 --- a/doc/architecture/PROJECT-LAYOUT.md +++ b/doc/architecture/PROJECT-LAYOUT.md @@ -2,13 +2,13 @@ ## Executive Summary -ProxySQL is a database proxy server written in C++ that provides protocol-aware proxying for MySQL and PostgreSQL databases. The system implements a multi-threaded, event-driven architecture with connection pooling, query routing, and monitoring. Built with C++11/17, it uses a modular design with separation between protocol handlers, session management, and administrative interfaces, backed by an embedded SQLite3 configuration database. +ProxySQL is a database proxy server written in C++ providing protocol-aware proxying for MySQL and PostgreSQL databases. It implements a multi-threaded, event-driven architecture with connection pooling, query routing, and monitoring. Built with C++11/17, it uses modular design separating protocol handlers, session management, and administrative interfaces, backed by embedded SQLite3 configuration. ## Key Components -- **Dual-protocol architecture**: MySQL and PostgreSQL wire protocol implementations -- **Multi-threaded design**: Separate worker threads for connections, admin, monitoring, and clustering -- **Embedded configuration**: SQLite3-based three-tier configuration system (Disk → Memory → Runtime) +- **Dual protocols**: MySQL and PostgreSQL wire protocol implementations +- **Multi-threaded**: Separate worker threads for connections, admin, monitoring, and clustering +- **Configuration**: SQLite3 three-tier system (Disk → Memory → Runtime) - **Performance features**: Lock-free structures, connection pooling, query caching - **Enterprise features**: Clustering, monitoring, REST API, and Prometheus metrics @@ -113,28 +113,28 @@ graph LR ### Code Organization ``` https://github.com/sysown/proxysql/tree/v3.0.agentics/ -├── src/ # Main entry points (4 core files) -│ ├── main.cpp # Primary application entry, thread initialization -│ ├── SQLite3_Server.cpp # Embedded configuration database +├── src/ # Main entry points (4 files) +│ ├── main.cpp # Application entry, thread initialization +│ ├── SQLite3_Server.cpp # Configuration database │ ├── proxy_tls.cpp # TLS/SSL implementation -│ └── proxysql_global.cpp # Global variables and configuration +│ └── proxysql_global.cpp # Global variables, configuration │ ├── lib/ # Core library implementations (86 .cpp files) │ ├── MySQL_*.cpp # MySQL protocol & management (20+ files) -│ │ ├── MySQL_Protocol.cpp # Wire protocol implementation -│ │ ├── MySQL_Session.cpp # Client session handling +│ │ ├── MySQL_Protocol.cpp # Wire protocol +│ │ ├── MySQL_Session.cpp # Session handling │ │ ├── MySQL_HostGroups_Manager.cpp # Backend management │ │ └── MySQL_Monitor.cpp # Health monitoring │ │ │ ├── PgSQL_*.cpp # PostgreSQL protocol & management (15+ files) -│ │ ├── PgSQL_Protocol.cpp # Wire protocol implementation -│ │ ├── PgSQL_Session.cpp # Client session handling -│ │ └── PgSQL_Authentication.cpp # SASL/SCRAM support +│ │ ├── PgSQL_Protocol.cpp # Wire protocol v3 +│ │ ├── PgSQL_Session.cpp # Session handling +│ │ └── PgSQL_Authentication.cpp # SASL/SCRAM │ │ │ ├── ProxySQL_Admin*.cpp # Administrative interface (10+ files) -│ │ ├── ProxySQL_Admin.cpp # Main admin implementation -│ │ ├── ProxySQL_Admin_Stats.cpp # Statistics collection -│ │ └── ProxySQL_RESTAPI_Server.cpp # REST API endpoint +│ │ ├── ProxySQL_Admin.cpp # Admin implementation +│ │ ├── ProxySQL_Admin_Stats.cpp # Statistics +│ │ └── ProxySQL_RESTAPI_Server.cpp # REST API │ │ │ ├── Base_*.cpp # Base infrastructure (5+ files) │ ├── Query_*.cpp # Query processing & caching @@ -156,8 +156,8 @@ https://github.com/sysown/proxysql/tree/v3.0.agentics/ │ └── ... # 17 more dependencies │ └── test/ # Test infrastructure - ├── tap/tests/ # 220+ TAP test files - ├── cluster/ # Cluster functionality tests + ├── tap/tests/ # 220+ TAP tests + ├── cluster/ # Cluster tests └── PrepStmt/ # Prepared statement tests ``` @@ -277,11 +277,11 @@ graph LR └─────────────────┘ ``` -### Port Mapping and Interfaces -- **6033**: MySQL/PostgreSQL client connections (main proxy port) -- **6032**: Admin interface (MySQL-compatible protocol) -- **6080**: REST API endpoint (HTTP/JSON) -- **6070**: Web UI interface (optional) +### Port Mapping +- **6033**: MySQL/PostgreSQL client connections +- **6032**: Admin interface (MySQL protocol) +- **6080**: REST API (HTTP/JSON) +- **6070**: Web UI (optional) ### Docker Deployment Scenarios ``` @@ -317,10 +317,10 @@ INSERT INTO mysql_query_rules ( ``` ### Health Monitoring -- **Automatic health checks**: Configurable intervals +- **Health checks**: Configurable intervals - **Connection verification**: `monitor_ping_interval` -- **Replication lag detection**: For read/write splitting -- **Automatic shunning**: Remove unhealthy backends +- **Replication lag detection**: Read/write splitting +- **Automatic shunning**: Removes unhealthy backends ## Performance and Scaling Architecture @@ -359,10 +359,10 @@ graph TD style BE fill:#e1f5fe ``` -### Caching Architecture -- **Query Cache**: Configurable TTL, pattern-based +### Caching +- **Query Cache**: TTL and pattern-based - **Prepared Statement Cache**: Metadata caching -- **Connection Attributes Cache**: Reduces authentication overhead +- **Connection Attributes Cache**: Reduced authentication overhead ## Testing and Quality Assurance @@ -411,11 +411,11 @@ mysql -h127.0.0.1 -P6032 -uadmin -padmin ## Architecture Features -1. **Protocol-Aware Proxy**: Understands and manipulates database protocols -2. **Runtime Reconfiguration**: Configuration changes without connection drops -3. **Load Balancing**: Weight-based, least-connections, round-robin algorithms -4. **Query Routing**: Regex-based rules, query rewriting, caching -5. **Integration**: Prometheus metrics, REST API, cluster support -6. **Security**: SSL/TLS, SQL injection detection, authentication plugins -7. **Performance**: Lock-free statistics, connection pooling, query cache -8. **High Availability**: Automatic failover, read/write splitting, replication lag detection \ No newline at end of file +1. **Protocol-Aware**: Understands and manipulates database protocols +2. **Runtime Reconfiguration**: Changes without connection drops +3. **Load Balancing**: Weight-based, least-connections, round-robin +4. **Query Routing**: Regex rules, rewriting, caching +5. **Integration**: Prometheus, REST API, clustering +6. **Security**: SSL/TLS, SQL injection detection, authentication +7. **Performance**: Lock-free statistics, connection pooling, caching +8. **High Availability**: Failover, read/write splitting, lag detection \ No newline at end of file diff --git a/doc/architecture/VISUAL-GUIDE.md b/doc/architecture/VISUAL-GUIDE.md index 13e83173f..d654a02c4 100644 --- a/doc/architecture/VISUAL-GUIDE.md +++ b/doc/architecture/VISUAL-GUIDE.md @@ -104,42 +104,42 @@ graph TB ``` proxysql/ ├── src/ [Main Entry Points - 4 files] -│ ├── main.cpp # Application entry, thread initialization -│ ├── SQLite3_Server.cpp # Embedded config database server +│ ├── main.cpp # Application entry, thread init +│ ├── SQLite3_Server.cpp # Config database server │ ├── proxy_tls.cpp # TLS/SSL implementation -│ └── proxysql_global.cpp # Global variables and config +│ └── proxysql_global.cpp # Global variables, config │ ├── lib/ [Core Libraries - 86+ files] │ ├── [MySQL Components - 25+ files] -│ │ ├── MySQL_Session.cpp # Client session management -│ │ ├── MySQL_Protocol.cpp # Wire protocol implementation -│ │ ├── MySQL_HostGroups_Manager.cpp # Backend server management +│ │ ├── MySQL_Session.cpp # Session management +│ │ ├── MySQL_Protocol.cpp # Wire protocol +│ │ ├── MySQL_HostGroups_Manager.cpp # Backend management │ │ ├── MySQL_Monitor.cpp # Health monitoring -│ │ ├── MySQL_Authentication.cpp # Auth methods (native, sha256, etc) -│ │ ├── MySQL_Query_Processor.cpp # Query routing logic +│ │ ├── MySQL_Authentication.cpp # Auth methods +│ │ ├── MySQL_Query_Processor.cpp # Query routing │ │ ├── MySQL_Query_Cache.cpp # Result caching -│ │ ├── MySQL_Thread.cpp # Thread pool management -│ │ ├── MySQL_Logger.cpp # Query/error logging -│ │ ├── MySQL_PreparedStatement.cpp # PS protocol handling +│ │ ├── MySQL_Thread.cpp # Thread pool +│ │ ├── MySQL_Logger.cpp # Logging +│ │ ├── MySQL_PreparedStatement.cpp # Prepared statements │ │ └── MySQL_Variables.cpp # Session variables │ │ │ ├── [PostgreSQL Components - 20+ files] -│ │ ├── PgSQL_Session.cpp # Client session management +│ │ ├── PgSQL_Session.cpp # Session management │ │ ├── PgSQL_Protocol.cpp # Wire protocol v3 │ │ ├── PgSQL_HostGroups_Manager.cpp # Backend management │ │ ├── PgSQL_Monitor.cpp # Health checks -│ │ ├── PgSQL_Authentication.cpp # SASL/SCRAM support +│ │ ├── PgSQL_Authentication.cpp # SASL/SCRAM │ │ ├── PgSQL_Query_Processor.cpp # Query routing │ │ ├── PgSQL_Query_Cache.cpp # Result caching │ │ ├── PgSQL_Thread.cpp # Thread management │ │ └── PgSQL_Logger.cpp # Logging │ │ │ ├── [Base Infrastructure - 10+ files] -│ │ ├── Base_Session.cpp # Template base for sessions -│ │ ├── Base_Thread.cpp # Common threading -│ │ ├── Base_HostGroups_Manager.cpp # Template base for HG -│ │ ├── Query_Processor.cpp # Common query processing -│ │ └── Query_Cache.cpp # Common caching logic +│ │ ├── Base_Session.cpp # Session template base +│ │ ├── Base_Thread.cpp # Threading base +│ │ ├── Base_HostGroups_Manager.cpp # HostGroups base +│ │ ├── Query_Processor.cpp # Query processing base +│ │ └── Query_Cache.cpp # Caching base │ │ │ ├── [Admin & Monitoring - 15+ files] │ │ ├── ProxySQL_Admin.cpp # Admin interface (6032) @@ -150,25 +150,25 @@ proxysql/ │ │ └── ProxySQL_Config.cpp # Configuration management │ │ │ └── [Supporting Libraries] -│ ├── Standard_Query_Cache.cpp # Query cache implementation -│ ├── MySQL_ResultSet.cpp # Result set handling +│ ├── Standard_Query_Cache.cpp # Query cache +│ ├── MySQL_ResultSet.cpp # Result sets │ ├── network.cpp # Network utilities -│ ├── debug.cpp # Debug/logging utilities -│ └── libproxysql.a # Compiled static library (340MB+) +│ ├── debug.cpp # Debug utilities +│ └── libproxysql.a # Static library (340MB+) │ ├── include/ [Header Files - 89+ files] │ ├── [Protocol Headers] -│ │ ├── MySQL_Protocol.h # MySQL protocol definitions -│ │ ├── PgSQL_Protocol.h # PostgreSQL protocol defs -│ │ ├── mysql_connection.h # MySQL connection handling +│ │ ├── MySQL_Protocol.h # MySQL protocol +│ │ ├── PgSQL_Protocol.h # PostgreSQL protocol +│ │ ├── mysql_connection.h # MySQL connections │ │ └── pgsql_connection.h # PostgreSQL connections │ │ │ ├── [Core Headers] │ │ ├── proxysql.h # Main header -│ │ ├── proxysql_structs.h # Core data structures -│ │ ├── Base_Session.h # Session template base -│ │ ├── Base_Thread.h # Thread base class -│ │ └── Base_HostGroups_Manager.h # HG template base +│ │ ├── proxysql_structs.h # Data structures +│ │ ├── Base_Session.h # Session base +│ │ ├── Base_Thread.h # Thread base +│ │ └── Base_HostGroups_Manager.h # HostGroups base │ │ │ └── [Utility Headers] │ ├── btree_map.h # B-tree implementation @@ -191,13 +191,13 @@ proxysql/ │ └── [12 more libraries...] │ └── test/ [Test Infrastructure] - ├── tap/ # TAP test framework - │ └── tests/ # 220+ test files - │ ├── test_mysql_*.cpp # MySQL-specific tests + ├── tap/ # TAP framework + │ └── tests/ # 220+ tests + │ ├── test_mysql_*.cpp # MySQL tests │ ├── test_pgsql_*.cpp # PostgreSQL tests - │ └── test_admin_*.cpp # Admin interface tests + │ └── test_admin_*.cpp # Admin tests ├── cluster/ # Cluster tests - └── PrepStmt/ # Prepared statement tests + └── PrepStmt/ # Prepared statements ``` ## Class Hierarchy Diagrams @@ -757,12 +757,12 @@ sequenceDiagram ## Thread Architecture -### Multi-Threaded Processing Model +### Thread Processing Model ```mermaid graph TB subgraph "Main Thread" - MAIN["main()"] + MAIN[main()] INIT[Initialize] CONFIG[Load Config] START[Start Threads] @@ -865,7 +865,7 @@ graph LR ## Connection Pooling Architecture -### Advanced Pool Management +### Pool Management ```mermaid graph TB @@ -1515,7 +1515,7 @@ graph LR ## Performance Optimization Points -### Critical Path Optimizations +### Performance Optimizations ```mermaid graph LR