13 KiB

Raw Blame History

Project Layout Analysis - ProxySQL

⚠️ Important Notice: This documentation was generated by AI and may contain inaccuracies. It should be used as a starting point for exploration only. Always verify critical information against the actual source code.

Last AI Update: 2025-09-11 Status: NON-VERIFIED Maintainer: Rene Cannao

Executive Summary

ProxySQL is a database proxy server written in C++ providing protocol-aware proxying for MySQL and PostgreSQL databases. It implements a multi-threaded, event-driven architecture with connection pooling, query routing, and monitoring. Built with C++11/17, it uses modular design separating protocol handlers, session management, and administrative interfaces, backed by embedded SQLite3 configuration.

Key Components

Dual protocols: MySQL and PostgreSQL wire protocol implementations
Multi-threaded: Separate worker threads for connections, admin, monitoring, and clustering
Configuration: SQLite3 three-tier system (Disk → Memory → Runtime)
Performance features: Lock-free structures, connection pooling, query caching
Enterprise features: Clustering, monitoring, REST API, and Prometheus metrics

System Architecture

graph TB
    subgraph "Client Layer"
        MC[MySQL Clients<br/>Port 6033]
        PC[PostgreSQL Clients<br/>Port 6033]
        AC[Admin Clients<br/>Port 6032]
    end
    
    subgraph "ProxySQL Core"
        MT[MySQL Thread Pool<br/>Protocol Handler]
        PT[PgSQL Thread Pool<br/>Protocol Handler]
        AT[Admin Thread<br/>Configuration & Stats]
        MON[Monitor Thread<br/>Health Checking]
        
        MC --> MT
        PC --> PT
        AC --> AT
    end
    
    subgraph "Core Services"
        QP[Query Processor<br/>Routing & Rewriting]
        QC[Query Cache<br/>Result Caching]
        HGM[HostGroups Manager<br/>Connection Pooling]
        AUTH[Authentication<br/>Multiple Methods]
        
        MT --> QP
        PT --> QP
        QP --> QC
        QP --> HGM
        MT --> AUTH
        PT --> AUTH
    end
    
    subgraph "Persistence Layer"
        SQL[(SQLite3<br/>Configuration DB)]
        DISK[Disk Config<br/>proxysql.cnf]
        STATS[Statistics Tables<br/>Runtime Metrics]
        
        AT --> SQL
        SQL --> DISK
        MON --> STATS
        SQL --> STATS
    end
    
    subgraph "Backend Databases"
        MDB[(MySQL/MariaDB<br/>Backends)]
        PDB[(PostgreSQL<br/>Backends)]
        
        HGM --> MDB
        HGM --> PDB
    end
    
    style MC fill:#e1f5fe
    style PC fill:#c8e6c9
    style AC fill:#fff3e0
    style SQL fill:#f3e5f5

Threading Model and Data Flow

graph LR
    subgraph "Connection Lifecycle"
        NEW[New Connection] --> ACCEPT[Accept Thread]
        ACCEPT --> WORKER[Worker Thread<br/>Assigned]
        WORKER --> SESSION[Session Object<br/>Created]
        SESSION --> PROCESS[Query Processing]
        PROCESS --> BACKEND[Backend Connection<br/>from Pool]
        BACKEND --> EXECUTE[Execute on Backend]
        EXECUTE --> CACHE[Cache Results<br/>if eligible]
        CACHE --> RESPONSE[Return to Client]
    end
    
    subgraph "Configuration Flow"
        FILE[proxysql.cnf] --> LOAD[Load to Memory]
        LOAD --> RUNTIME[Apply to Runtime]
        RUNTIME --> PERSIST[Save to Disk]
        ADMIN[Admin Interface] --> RUNTIME
        CLUSTER[Cluster Sync] --> RUNTIME
    end
    
    subgraph "Monitoring Flow"
        CHECK[Health Check] --> STATUS[Backend Status]
        STATUS --> HOSTGROUP[Update HostGroup]
        HOSTGROUP --> ROUTING[Adjust Routing]
        STATS2[Collect Stats] --> EXPOSE[Expose Metrics]
        EXPOSE --> PROM[Prometheus/REST]
    end
    
    style NEW fill:#e8f5e8
    style FILE fill:#fff3e0
    style CHECK fill:#ffebee

Directory Structure Mapping

Code Organization

https://github.com/sysown/proxysql/tree/v3.0.agentics/
├── src/                        # Main entry points (4 files)
│   ├── main.cpp               # Application entry, thread initialization
│   ├── SQLite3_Server.cpp     # Configuration database
│   ├── proxy_tls.cpp          # TLS/SSL implementation
│   └── proxysql_global.cpp    # Global variables, configuration
│
├── lib/                        # Core library implementations (86 .cpp files)
│   ├── MySQL_*.cpp            # MySQL protocol & management (20+ files)
│   │   ├── MySQL_Protocol.cpp        # Wire protocol
│   │   ├── MySQL_Session.cpp         # Session handling
│   │   ├── MySQL_HostGroups_Manager.cpp # Backend management
│   │   └── MySQL_Monitor.cpp         # Health monitoring
│   │
│   ├── PgSQL_*.cpp            # PostgreSQL protocol & management (15+ files)
│   │   ├── PgSQL_Protocol.cpp        # Wire protocol v3
│   │   ├── PgSQL_Session.cpp         # Session handling
│   │   └── PgSQL_Authentication.cpp  # SASL/SCRAM
│   │
│   ├── ProxySQL_Admin*.cpp    # Administrative interface (10+ files)
│   │   ├── ProxySQL_Admin.cpp        # Admin implementation
│   │   ├── ProxySQL_Admin_Stats.cpp  # Statistics
│   │   └── ProxySQL_RESTAPI_Server.cpp # REST API
│   │
│   ├── Base_*.cpp             # Base infrastructure (5+ files)
│   ├── Query_*.cpp            # Query processing & caching
│   └── libproxysql.a          # Compiled static library (340MB+)
│
├── include/                    # Header files (89 .h files)
│   ├── MySQL_*.h              # MySQL protocol headers
│   ├── PgSQL_*.h              # PostgreSQL protocol headers
│   ├── proxysql_*.h           # Core infrastructure headers
│   └── btree*.h               # Performance data structures
│
├── deps/                       # External dependencies (23 libraries)
│   ├── mariadb-client-library/ # MySQL/MariaDB connector
│   ├── postgresql/             # PostgreSQL client library
│   ├── sqlite3/                # Embedded database
│   ├── libev/                  # Event loop library
│   ├── jemalloc/               # Memory allocator
│   ├── prometheus-cpp/         # Metrics library
│   └── ...                     # 17 more dependencies
│
└── test/                       # Test infrastructure
    ├── tap/tests/              # 220+ TAP tests
    ├── cluster/                # Cluster tests
    └── PrepStmt/               # Prepared statement tests

Build System and Artifacts

graph TB
    subgraph "Build Targets"
        BT1[make → Standard build]
        BT2[make debug → Debug build]
        BT3[make build_deps → Build dependencies]
        BT4[make build_tap_test_debug → Test framework]
    end
    
    subgraph "Build Artifacts"
        BA1[src/proxysql → Main binary]
        BA2[lib/libproxysql.a → Static library]
        BA3[lib/obj/*.o → Object files]
        BA4[out/production/proxysql → Production binary]
        BA5[out/test/proxysql → Test binary]
    end
    
    subgraph "Dependencies Built"
        DB1[deps/mariadb-client-library/]
        DB2[deps/postgresql/]
        DB3[deps/sqlite3/]
        DB4[deps/jemalloc/]
        DB5[deps/libev/]
    end
    
    BT1 --> BA1
    BT1 --> BA2
    BT2 --> BA4
    BT3 --> DB1
    BT3 --> DB2
    BT3 --> DB3
    BT4 --> BA5
    
    style BT1 fill:#e8f5e8
    style BT2 fill:#fff3e0
    style BA1 fill:#e1f5fe

Module Dependencies and Communication

graph LR
    subgraph "Protocol Layer"
        MYSQL[MySQL Protocol]
        PGSQL[PgSQL Protocol]
    end
    
    subgraph "Session Management"
        MSESS[MySQL Sessions]
        PSESS[PgSQL Sessions]
        BSESS[Base Session]
        
        MSESS --> BSESS
        PSESS --> BSESS
    end
    
    subgraph "Backend Management"
        MHGM[MySQL HostGroups]
        PHGM[PgSQL HostGroups]
        BHGM[Base HostGroups]
        POOL[Connection Pool]
        
        MHGM --> BHGM
        PHGM --> BHGM
        BHGM --> POOL
    end
    
    subgraph "Core Services"
        QP2[Query Processor]
        QC2[Query Cache]
        AUTH2[Authentication]
        MON2[Monitor Service]
    end
    
    subgraph "Infrastructure"
        SQLITE[SQLite3 Config]
        EVENT[libev Events]
        MEM[jemalloc Memory]
        METRICS[Prometheus Metrics]
    end
    
    MYSQL --> MSESS
    PGSQL --> PSESS
    MSESS --> QP2
    PSESS --> QP2
    QP2 --> QC2
    QP2 --> MHGM
    QP2 --> PHGM
    MON2 --> MHGM
    MON2 --> PHGM
    
    MSESS --> EVENT
    PSESS --> EVENT
    POOL --> MEM
    MON2 --> METRICS
    
    style MYSQL fill:#e1f5fe
    style PGSQL fill:#c8e6c9
    style SQLITE fill:#f3e5f5

Configuration and Deployment Architecture

Three-Tier Configuration System

┌─────────────────┐
│   Disk Layer    │  ← proxysql.cnf, persistent storage
├─────────────────┤
│  Memory Layer   │  ← Loaded configuration, staging area
├─────────────────┤
│ Runtime Layer   │  ← Active configuration, live system
└─────────────────┘

Port Mapping

6033: MySQL/PostgreSQL client connections
6032: Admin interface (MySQL protocol)
6080: REST API (HTTP/JSON)
6070: Web UI (optional)

Docker Deployment Scenarios

docker/
├── 1backend/                   # Single backend configuration
├── 5backends-replication/      # Multi-backend with replication
├── images/                     # Container image definitions
│   ├── mysql/                 # MySQL test containers
│   └── proxysql/              # ProxySQL containers
└── scenarios/                  # Testing scenarios

Key Configuration and Operational Patterns

Runtime Configuration Management

-- Admin interface SQL commands (port 6032)
INSERT INTO mysql_servers(...) VALUES(...);    -- Add backend
LOAD MYSQL SERVERS TO RUNTIME;                 -- Apply changes
SAVE MYSQL SERVERS TO DISK;                    -- Persist

-- Monitor configuration
UPDATE global_variables SET variable_value=...;
LOAD ADMIN VARIABLES TO RUNTIME;

Query Routing Rules

-- Define routing rules
INSERT INTO mysql_query_rules (
    rule_id, match_pattern, destination_hostgroup
) VALUES (1, '^SELECT.*FOR UPDATE', 0);       -- Write queries to HG 0

Health Monitoring

Health checks: Configurable intervals
Connection verification: monitor_ping_interval
Replication lag detection: Read/write splitting
Automatic shunning: Removes unhealthy backends

Performance and Scaling Architecture

Connection Pooling Strategy

graph TD
    subgraph "Client Connections"
        C1[Client 1]
        C2[Client 2]
        C3[Client N]
    end
    
    subgraph "ProxySQL Connection Multiplexing"
        FE[Frontend Connections<br/>1000s possible]
        MUX[Multiplexer]
        BE[Backend Pool<br/>10s-100s connections]
    end
    
    subgraph "Backend Servers"
        HG0[HostGroup 0<br/>Writers]
        HG1[HostGroup 1<br/>Readers]
        HG2[HostGroup 2<br/>Analytics]
    end
    
    C1 --> FE
    C2 --> FE
    C3 --> FE
    FE --> MUX
    MUX --> BE
    BE --> HG0
    BE --> HG1
    BE --> HG2
    
    style FE fill:#e8f5e8
    style MUX fill:#fff3e0
    style BE fill:#e1f5fe

Caching

Query Cache: TTL and pattern-based
Prepared Statement Cache: Metadata caching
Connection Attributes Cache: Reduced authentication overhead

Testing and Quality Assurance

Test Infrastructure Layers

Unit Tests: Component-level testing
TAP Tests: 220+ Test Anything Protocol tests
Integration Tests: Full system scenarios
Cluster Tests: Multi-node configurations
Fuzzing Tests: AFL-based security testing

Test Execution

# Run TAP tests
make build_tap_test_debug
cd test/tap/tests
./run_tests.sh

# Run specific test suite
./test_mysql_connect-t
./test_pgsql_authentication-t

Build and Development Workflow

Development Setup

# Clone and build with dependencies
git clone https://github.com/sysown/proxysql.git
cd proxysql
make build_deps
make debug

# Run with custom config
./src/proxysql -f -c proxysql.cnf

# Connect to admin interface
mysql -h127.0.0.1 -P6032 -uadmin -padmin

Debug Build Features

Full debugging symbols
Assertion checks enabled
Memory leak detection
Core dump generation
Verbose logging

Architecture Features

Protocol-Aware: Understands and manipulates database protocols
Runtime Reconfiguration: Changes without connection drops
Load Balancing: Weight-based, least-connections, round-robin
Query Routing: Regex rules, rewriting, caching
Integration: Prometheus, REST API, clustering
Security: SSL/TLS, SQL injection detection, authentication
Performance: Lock-free statistics, connection pooling, caching
High Availability: Failover, read/write splitting, lag detection

13 KiB Raw Blame History