proxysql/test/infra

ProxySQL Unified CI Infrastructure

This directory contains the self-contained infrastructure configurations and control scripts for the ProxySQL Unified CI System. It enables high-concurrency, isolated test environments using Docker-outside-of-Docker (DooD) and Pure Network Isolation.

---

0. Pre-requirement: Building the CI Base Image

The ProxySQL control plane and test runner use a standardized toolbelt image: proxysql-ci-base:latest. This image must be built locally before starting any infrastructure:

cd test/infra/docker-base
docker build --network host -t proxysql-ci-base:latest .
cd ../../../

Note: Using --network host is recommended if you encounter DNS resolution hangs during the build process.

---

0.1. Building ProxySQL (Debug Mode)

Some tests require ProxySQL to be compiled in debug mode (e.g., for LOAD DEBUG FROM DISK support). If you encounter errors like near "LOAD": syntax error when the test runner tries to execute LOAD DEBUG FROM DISK, you need to rebuild ProxySQL in debug mode:

export PROXYSQLGENAI=1
make -j$(nproc) build_deps && make -j$(nproc) debug && make -j$(nproc) build_deps && make -j$(nproc) build_tap_test_debug

This will:

1. Build all dependencies
2. Compile ProxySQL with debug symbols and the DEBUG module enabled
3. Rebuild dependencies (if needed for debug mode)
4. Build TAP test executables in debug mode

Note: After rebuilding, restart your infrastructure with a fresh INFRA_ID to use the new binary.

---

1. Core Concepts

The Unified CI system is designed around three pillars:

1. Isolation: Every test run uses a unique INFRA_ID. This ID namespaces all Docker containers and networks, allowing multiple developers or CI jobs to run on the same host without port collisions.
2. Parity: The exact same scripts and configurations are used in Jenkins, local development, and by AI agents.
3. Automation: Provisioning of backends (MySQL, MariaDB, etc.) and their registration into ProxySQL is fully scripted via dynamic SQL templates.

---

2. Manual Execution Guide

The infrastructure management and test execution are strictly separated.

Step 1: Global Setup

IMPORTANT: INFRA_ID must be unique for each test run to avoid conflicts. Use a timestamp to ensure uniqueness:

export WORKSPACE=$(pwd)
export INFRA_ID="test-$(date +%s)"   # Unique ID using timestamp
export TAP_GROUP="mysql84-g1"
source test/infra/common/env.sh

Why unique INFRA_ID?

• Prevents port collisions when multiple tests run on the same host
• Ensures clean isolation between test environments
• Avoids conflicts with existing Docker containers and networks

Step 2: Start ProxySQL & Backends

You can use the helper script to automatically start all required components for a group:

./test/infra/control/ensure-infras.bash

This script will start ProxySQL and then iterate through the infras.lst of the specified group.

Step 3: Run TAP Tests

The test runner script will verify that all required containers are running before starting.

./test/infra/control/run-tests-isolated.bash

Step 4: Teardown

Manual cleanup of all components.

# Destroy backends for the current group
./test/infra/control/stop-proxysql-isolated.bash

---

3. Test Runner & Infrastructure Verification

The run-tests-isolated.bash script acts as a validator and execution orchestrator.

The infras.lst Mechanism

Every TAP test group (in test/tap/groups/<group_name>) defines its required backend environments using an infras.lst file.

• Who reads it?: Both ensure-infras.bash (to start them) and run-tests-isolated.bash (to verify them).
• Order Importance: In multi-infrastructure groups, the order of entries is critical. Shared ProxySQL users (e.g., testuser, root) are registered using INSERT OR IGNORE. The first infrastructure in the list that defines a shared user will set its default_hostgroup.
• Subgroups: If a group has a suffix (e.g., legacy-g1), the system automatically strips it to find the base group definition (legacy).

Safety-First Initialization

The docker-compose-init.bash scripts implement a strict non-destructive policy:

• They will refuse to start if the target data or log directories on the host are not empty.
• This prevents accidental data loss or inconsistent test states.
• You must run docker-compose-destroy.bash within the infra folder or manually clean ci_infra_logs/${INFRA_ID}/ before re-initializing.

---

4. Environment Variables Reference

Variable	Default	Description
INFRA_ID	dev-$USER	Required. Unique namespace for Docker containers/networks. Must be unique - use a timestamp for isolation: export INFRA_ID="test-$(date +%s)"
WORKSPACE	Repo Root	Root path of the ProxySQL repository.
TAP_GROUP	(none)	Run a specific group defined in test/tap/groups/groups.json.
TEST_PY_TAP_INCL	(none)	Filter tests within the group (regex matching test names).
SKIP_CLUSTER_START	0	Set to 1 to bypass starting additional ProxySQL nodes.
SKIP_PROXYSQL	0	Set to 1 in a group's env.sh to skip ProxySQL and all backend infrastructure. When enabled, ensure-infras.bash exits immediately and run-tests-isolated.bash runs tests directly on the host without Docker. Used by the unit-tests group.
PROXY_DATA_DIR_HOST	(dynamic)	Host path for ProxySQL persistent data.
COVERAGE	0	Enable code coverage collection.
TAP_USE_NOISE	0	Enable noise injection for race condition testing.

---

5. Examples & Use Cases

Example 1: Run a single test in a MySQL 8.4 environment

If you are developing a fix and only want to run one specific test:

export INFRA_ID="fix-123-$(date +%s)"  # Unique ID with timestamp
export TAP_GROUP="mysql84-g1"
export TEST_PY_TAP_INCL="admin_various_commands-t"
export SKIP_CLUSTER_START=1

./test/infra/control/ensure-infras.bash
./test/infra/control/run-tests-isolated.bash

Example 2: Run all MariaDB 10 tests in parallel with another run

export INFRA_ID="mariadb-test-$(date +%s)"  # Unique ID with timestamp
export TAP_GROUP="mariadb10-g1"
./test/infra/control/ensure-infras.bash
./test/infra/control/run-tests-isolated.bash

Example 3: Debugging a failing PostgreSQL test in the Legacy group

export INFRA_ID="debug-legacy-$(date +%s)"  # Unique ID with timestamp
export TAP_GROUP="legacy-g1"
export TEST_PY_TAP_INCL="pgsql-.*" # Run only PGSQL tests in legacy
./test/infra/control/ensure-infras.bash
./test/infra/control/run-tests-isolated.bash

---

6. Advanced: Custom Test Groups

For rapid iteration, you can create a temporary subgroup to target exactly the infrastructure and tests you need.

How to create a custom subgroup (e.g., mysql84-g5)

1. Define Infrastructure: The system will look for test/tap/groups/mysql84/infras.lst (because it strips the -g5 suffix).
2. Add Tests to groups.json: Add your specific tests to the new group in test/tap/groups/groups.json:

   "my_new_test-t": [ "mysql84-g5" ]
   

3. Run:

   export TAP_GROUP="mysql84-g5"
   ./test/infra/control/ensure-infras.bash
   ./test/infra/control/run-tests-isolated.bash
   

---

6.1. Infrastructure-Free Groups (Unit Tests)

Some test groups do not require ProxySQL or any backend infrastructure. These groups set SKIP_PROXYSQL=1 in their env.sh, which causes:

• ensure-infras.bash to exit immediately (no Docker containers started)
• run-tests-isolated.bash to run tests directly on the host (no Docker test runner)

The unit-tests group is the primary example. Unit tests link against libproxysql.a with stub globals and run as standalone binaries.

Running Unit Tests

# Build unit tests
cd test/tap/tests/unit && make && cd -

# Run via the multi-group runner (recommended)
TAP_GROUPS="unit-tests-g1" ./test/infra/control/run-multi-group.bash

# Or run directly
export TAP_GROUP="unit-tests-g1"
./test/infra/control/ensure-infras.bash   # exits immediately (SKIP_PROXYSQL=1)
./test/infra/control/run-tests-isolated.bash

Creating Your Own Infrastructure-Free Group

1. Create a group directory: test/tap/groups/my-group/
2. Add env.sh with export SKIP_PROXYSQL=1
3. Add your tests to groups.json under my-group-g1
4. No infras.lst needed

---

7. ProxySQL Cluster Management

By default, the infrastructure starts a single ProxySQL node. For cluster-specific testing, you can enable additional nodes.

Starting a Cluster

# 1. Start the main ProxySQL node
./test/infra/control/start-proxysql-isolated.bash

# 2. Start additional nodes (default: 9 nodes)
export PROXYSQL_CLUSTER_NODES=3
./test/infra/control/cluster_start.bash

# 3. Initialize the cluster
./test/infra/control/cluster_init.bash

---

8. Multi-Group Parallel Execution

The run-multi-group.bash script enables running multiple TAP groups in parallel with proper isolation and resource management.

Basic Usage

RUN_ID="test-$(date +%s)" \
TAP_GROUPS="legacy-g1 legacy-g2 legacy-g3 mysql84-g1" \
./test/infra/control/run-multi-group.bash

Configuration Options

Variable	Default	Description
RUN_ID	(timestamp)	Unique identifier for the multi-group run.
TAP_GROUPS	(required)	Space-separated list of TAP groups to run.
PARALLEL_JOBS	2	Maximum number of groups running in parallel.
TIMEOUT_MINUTES	60	Hard timeout per group in minutes.
EXIT_ON_FIRST_FAIL	0	Stop all groups on first failure if set to 1.
AUTO_CLEANUP	0	Automatically cleanup successful groups if set to 1.
STAGGER_DELAY	5	Seconds between group startups to prevent resource contention.
COVERAGE	0	Enable code coverage collection if set to 1.
TAP_USE_NOISE	0	Enable noise injection for race condition testing if set to 1.

Output Location

• Individual group logs: ci_infra_logs/multi-group-{RUN_ID}/{group}.log
• Combined coverage report: ci_infra_logs/multi-group-{RUN_ID}/coverage-report/

---

9. Code Coverage Collection

When running tests with COVERAGE=1, the system collects code coverage data from ProxySQL.

Requirements

• ProxySQL must be compiled with COVERAGE=1 (adds --coverage flags)
• fastcov and genhtml must be available in the proxysql-ci-base container

Usage

# Single group with coverage
COVERAGE=1 ./test/infra/control/run-tests-isolated.bash

# Multi-group with coverage (reports are combined)
COVERAGE=1 RUN_ID="cov-$(date +%s)" TAP_GROUPS="legacy-g1 legacy-g2" ./test/infra/control/run-multi-group.bash

Output

• Individual reports: ci_infra_logs/{INFRA_ID}/coverage-report/
• Combined report: ci_infra_logs/multi-group-{RUN_ID}/coverage-report/

---

10. Noise Injection Testing

Noise injection helps detect race conditions and deadlocks by introducing random delays and stress during test execution.

Usage

# Enable noise injection for a single group
TAP_USE_NOISE=1 ./test/infra/control/run-tests-isolated.bash

# Enable noise injection for multi-group run
TAP_USE_NOISE=1 RUN_ID="noise-$(date +%s)" TAP_GROUPS="legacy-g1 legacy-g2" ./test/infra/control/run-multi-group.bash

For more details, see test/tap/NOISE_TESTING.md.

---

11. Troubleshooting

• "Directory Not Empty": Run ./test/infra/control/stop-proxysql-isolated.bash with the same INFRA_ID to cleanup, or manually delete the folder in ci_infra_logs/.
• Permission Denied: The system uses sudo for log directory management. Ensure your user has sudo privileges.
• Container Crash: Check logs in ci_infra_logs/${INFRA_ID}/${COMPOSE_PROJECT}/${CONTAINER_NAME}/.