proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	39840b5393	Exclude /usr and deps from coverage reports fastcov was including system headers (/usr/include/c++/...) and vendored dependencies (deps/json/json.hpp) in coverage output. Use --exclude instead of --include since project source files have bare filenames in gcov output (no path prefix to match against), while system/deps paths are always absolute.	1 month ago
Rene Cannao	5a8b6e9a89	Fix gcov coverage collection for multi-group CI runs Three problems caused machine stalls and 0% coverage in multi-group runs: 1. All test-runner containers wrote TAP test .gcda files to the shared ${WORKSPACE}, causing concurrent corruption and memory bloat when multiple gcov processes fought over the same files. 2. Each container independently ran fastcov -j$(nproc) in its EXIT trap, spawning hundreds of concurrent gcov processes across all containers. 3. GCOV_PREFIX_STRIP=3 placed .gcda at proxysql/lib/obj/ but .gcno was copied to lib/obj/ — path mismatch meant gcov couldn't correlate them. Additionally, the --include filter expected full workspace paths but gcov records bare filenames, so all results were filtered out. Changes: - Set GCOV_PREFIX=/gcov/tap on test-runner containers so TAP .gcda files go to per-INFRA_ID /gcov/tap/ instead of shared workspace. - Pass MULTI_GROUP=1 from run-multi-group.bash to test-runners. When set, the EXIT trap copies .gcno adjacent to .gcda (handling path offset from GCOV_PREFIX_STRIP) but skips fastcov/genhtml. - Add centralized coverage collection in run-multi-group.bash: processes each group's /gcov directory sequentially with fastcov -j4 after all groups finish, then combines the .info files. - Remove --include filter from fastcov (source paths are bare filenames, not full paths, so the filter rejected everything). - Standalone mode (no MULTI_GROUP) is unchanged.	1 month ago
Rene Cannao	83c3e76506	Fix libparser.so loading for CI test binaries TAP test binaries link libcpp_dotenv.so which transitively needs libparser.so. The RUNPATH baked into libcpp_dotenv.so contains stale paths from the original build, so the dynamic linker can't find it. - Add LD_LIBRARY_PATH for the SKIP_PROXYSQL (unit test) host path - Add TAP_DEPS_PATH to env-isolated.bash so proxysql-tester.py constructs the correct LD_LIBRARY_PATH inside Docker containers	1 month ago
Rene Cannao	c7647c7387	Fix coverage: generate separate reports and combine with fastcov -C Coverage data comes from two sources that may overlap: /gcov — ProxySQL daemon execution (via GCOV_PREFIX) WORKSPACE — TAP test execution (.gcda next to binaries) Both may cover the same code (e.g. libproxysql.a exercised by both ProxySQL runtime and TAP test harness). Previously we copied .gcda files between directories, which either overwrote (cp -f) or kept only the first version (cp -n) instead of combining counters. Now generates two separate .info files (proxysql.info + tap_tests.info) and combines them with fastcov -C, which correctly sums execution counters for overlapping source files.	1 month ago
Rene Cannao	8ca54a9def	Fix coverage: don't overwrite ProxySQL's gcda files with stale copies The coverage collection copied ALL .gcno and .gcda files from the workspace to /gcov using cp -f. But ProxySQL's .gcda files (with real execution data from the Docker container) were already in /gcov via GCOV_PREFIX. The cp -f overwrote them with the host's stale .gcda files (zero or outdated execution counts). This caused lib/MySQLFFTO.cpp and lib/PgSQLFFTO.cpp (and likely other ProxySQL library files) to show 0% coverage despite the FFTO tests running successfully and the .gcda files having real data. Fix: copy .gcno files with cp -f (always needed for fastcov), but copy .gcda files with cp -n (no-clobber) so ProxySQL's existing execution data in /gcov is preserved. TAP test .gcda files that don't exist in /gcov yet are still copied.	1 month ago
Rene Cannao	fb9ff4476e	Include TAP test coverage: copy .gcda files from source tree to /gcov TAP test binaries are compiled with gcov but their .gcda files are written to the source tree (next to the binaries), not to /gcov. The coverage collection only scanned /gcov, missing all TAP test coverage data. Now copies both .gcno and .gcda files from the workspace into /gcov before running fastcov, so TAP test coverage is included in reports.	1 month ago
Rene Cannao	8705685f83	Address PR review comments: port collisions, gcov, loop bounds, fastcov - RESTAPI port offset changed from 6070+i to 7070+i to avoid collision with admin/mysql ports (6072 would collide with node4 admin) - Unset GCOV_PREFIX for background cluster nodes to prevent concurrent .gcda write corruption - Cap core-node and scheduler loops to min(NUM_NODES, 3) so smaller clusters don't reference non-existent nodes - TAP_CLUSTER_NODES now respects PROXYSQL_CLUSTER_NODES and SKIP_CLUSTER_START instead of hardcoding 9 - Fix fastcov flag: -i is invalid, use --include for source filtering	1 month ago
Rene Cannao	0590803889	Limit code coverage to proxysql source directories only Replace exclude-based filtering (-e /usr/include, deps/) with include-based filtering (-i) for include/, lib/, src/, test/. This avoids capturing coverage for system headers, vendored dependencies, and other non-proxysql code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 month ago
Rene Cannao	8c67073c15	Redirect coverage generation output to dedicated log file fastcov and genhtml produce verbose output that clutters the group test logs. Now redirected to coverage-generation.log inside the coverage report directory. Status messages still go to stdout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 month ago
Rene Cannao	879ebdc220	Fix startup order and add readiness check in test runner ensure-infras.bash: Move pre-proxysql hook (cluster setup) to run BEFORE backend infras start. Backends need ProxySQL and cluster to be fully ready because their docker-proxy-post.bash configures ProxySQL. Previous order was: ProxySQL → infras → cluster. New order: ProxySQL → cluster → infras (one by one) → setup hook. run-tests-isolated.bash: Add a readiness loop that waits up to 30s for ProxySQL admin port to be reachable from inside the test-runner container before running anything. Docker DNS resolution on newly created containers can take several seconds under load, causing "no route to host" (errno 113) errors on the first connection attempt. On failure, dumps DNS resolution diagnostics instead of silently proceeding. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 month ago
Rene Cannao	7df23de546	Remove error suppression from ProxySQL config dump queries Stop redirecting stderr to /dev/null and silently continuing on failure. If the config dump queries fail, we need to see the actual error and the test run should fail visibly, not silently proceed with a broken configuration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 month ago
Rene Cannao	4dfaedcb84	Revert "Skip missing unit test binaries instead of counting as failures" This reverts commit `75d324680d`.	1 month ago
Rene Cannao	75d324680d	Skip missing unit test binaries instead of counting as failures Unit tests in groups.json may reference binaries that haven't been compiled (they require a separate build step). Previously, not-found binaries were counted as test failures. Now they are skipped with a message, so the group passes if all compiled tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 month ago
Rene Cannao	6d90bc2a7e	Source group env.sh inside test runner container The group's env.sh (setting TEST_PY_* flags) was only sourced on the host. The Docker test runner container didn't see these values, so env-isolated.bash defaults took over (all tests disabled). Now source group env.sh before env-isolated.bash inside the container so group-specific settings are preserved. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 month ago
René Cannaò	3aab0c280f	Fix SKIP_PROXYSQL path: run test binaries directly instead of proxysql-tester.py proxysql-tester.py requires a ProxySQL admin connection and Docker-specific env vars (TESTS_LOGS_PATH, TAP_ADMINPORT, TAP_WORKDIR, etc.) that don't exist when SKIP_PROXYSQL=1. Instead, discover test binaries from groups.json and execute them directly on the host with a simple pass/fail summary. Addresses review feedback from CodeRabbit, Copilot, and Gemini on PR #5520.	1 month ago
René Cannaò	4c4736c73d	Add SKIP_PROXYSQL support for infrastructure-free test groups Unit tests run standalone against libproxysql.a with stubs and need no ProxySQL instance or backend databases. This adds a SKIP_PROXYSQL flag that test groups can set in their env.sh to bypass all Docker infrastructure setup and run tests directly on the host. Changes: - ensure-infras.bash: source group env.sh early; exit immediately when SKIP_PROXYSQL=1 - run-tests-isolated.bash: source group env.sh early; skip ProxySQL and backend verification; run proxysql-tester.py directly on host instead of inside a Docker container - New test/tap/groups/unit-tests/env.sh with SKIP_PROXYSQL=1 - groups.json: add 18 unit test binaries to unit-tests-g1 group - Documentation: SKIP_PROXYSQL in env var tables and new section for infrastructure-free groups in test/infra/README.md	1 month ago
René Cannaò	abd1da861f	Merge pull request #5470 from sysown/private/multi-group-runner Multi-group TAP test runner and ClickHouse test fixes	1 month ago
Rene Cannao	60b5aff5c4	Fix tests not running after infrastructure setup in multi-group runner The heredoc in run-multi-group.bash was causing bash to exit prematurely after ensure-infras.bash completed, without executing the test commands. This was due to shell expansion issues with the heredoc structure. Changes: - Replace heredoc with temp file approach in run-multi-group.bash - Add pre-proxysql hook call to ensure-infras.bash for cluster setup - Create legacy/pre-proxysql.bash for cluster node startup - Update mysql84/pre-proxysql.bash with cluster startup calls - Fix coverage collection to work correctly in parallel execution	1 month ago
Rene Cannao	a1042ce13f	Fix code coverage collection: mount gcov directory in test runner The test runner container needs access to the /gcov directory where ProxySQL writes its coverage data (.gcda files). Without this mount, the coverage collection script could only see .gcno files (compile-time) but not the runtime coverage data from ProxySQL. This fix adds the gcov volume mount to the test runner container, allowing it to merge coverage data from each ProxySQL instance.	1 month ago
Rene Cannao	be290c5aeb	Address PR review feedback on test infrastructure - Remove redundant host-side coverage collection in run-tests-isolated.bash (coverage is already collected via trap inside the container) - Initialize TAP_GROUP, group_has_tests, groups before try block in proxysql-tester.py to avoid UnboundLocalError on early exceptions - Differentiate skipped tests from missing tests in TAP_GROUP validation so intentionally skipped tests don't count as missing	1 month ago
Rene Cannao	86e86c8ce3	Fix parallel coverage collection and improve multi-group runner - Fix code coverage race condition: Each ProxySQL instance now writes .gcda files to its own directory using GCOV_PREFIX, preventing parallel groups from overwriting each other's coverage data - Add TAP_USE_NOISE documentation for noise injection testing - Replace random stagger delay with sequential delay (STAGGER_DELAY) to guarantee proper spacing between group startups - Update README with multi-group runner, coverage, and noise injection docs	1 month ago
Rene Cannao	0bc75cd589	Fix test validation and coverage HTML generation - Fix false positive "missing tests" errors when processing secondary workdirs (e.g., deprecate_eof_support). The validation now only checks tests that actually exist in the current workdir rather than all tests in groups.json for the TAP_GROUP. - Add --ignore-errors negative,source --synthesize-missing flags to genhtml calls in run-tests-isolated.bash to match the multi-group runner and prevent HTML generation failures on coverage data with negative branch counts.	1 month ago
Rene Cannao	20fc53e45c	Fix code coverage collection in multi-group test runner 1. Add set -e to run-multi-group.bash heredoc for proper error handling 2. Add EXIT trap in run-tests-isolated.bash to collect coverage on timeout/failure 3. Pass COVERAGE_MODE, COVERAGE_REPORT_DIR, and SCRIPT_DIR env vars to container 4. Fix fastcov invocation to use -e deps/ instead of restrictive -i filters 5. Add per-group HTML report generation 6. Run coverage combination in container for host without tools Coverage is now collected for: - include/ (189 header files) - lib/ (109 library files) - src/ (4 source files) The EXIT trap ensures coverage is collected even when tests timeout.	1 month ago
Rene Cannao	30620f37c4	Revert "Move coverage collection inside Docker container" This reverts commit `e571f08a45`.	1 month ago
Rene Cannao	e571f08a45	Move coverage collection inside Docker container Coverage collection now runs INSIDE the test-runner container immediately after tests complete (whether they succeed, fail, or timeout). This ensures: 1. Coverage data is collected even if the parent script times out 2. The .info file is generated inside the container before it exits 3. No dependency on external docker exec after container stops Also removes the external coverage collection code that was unreachable when timeout killed the subshell.	1 month ago
Rene Cannao	9f6ba655bf	Add optional code coverage support for isolated test runs	1 month ago
Rene Cannao	cf54bff262	Fix docker-proxy-post.bash to use absolute path for .env sourcing The scripts were using relative path [ -f .env ] which failed when called from test/infra/control/, causing all SQL template variables (PREFIX, WHG, RHG, etc.) to be empty. This resulted in query rules with NULL match_digest patterns and incorrect rule IDs.	1 month ago
Rene Cannao	2f8e63cef2	Address code review feedback for AI TAP group migration Fixes from PR #5461 review: 1. Fix BASE_GROUP derivation inconsistency in run-tests-isolated.bash - Changed from bash pattern matching `${TAP_GROUP%%-g[0-9]}` to sed - Now uses same pattern as ensure-infras.bash: `sed -E "s/[-_]g[0-9]+.//"` - Removed redundant BASE_GROUP reassignment 2. Update outdated comments in seed files - seed-mysql.sql and seed-pgsql.sql now correctly reference setup-infras.bash 3. Change INSERT OR IGNORE to INSERT OR REPLACE in mcp-config.sql - Ensures credentials are updated on reruns for deterministic state 4. Add shellcheck directives to env.sh files - Added `# shellcheck shell=bash` to ai/env.sh and ai-g1/env.sh 5. Add explicit validation for derived infrastructure names - setup-infras.bash now validates DEFAULT_MYSQL_INFRA and DEFAULT_PGSQL_INFRA - Provides clear error messages if infras.lst is misconfigured 6. Add WORKSPACE export to README examples - All runnable examples now include `export WORKSPACE=$(pwd)` Test verification: - ai-g1 group infrastructure setup works correctly - legacy-g2 group tests still pass (backward compatibility verified)	1 month ago
Rene Cannao	0b9806c46e	Migrate AI TAP group to Unified CI Infrastructure pattern This commit migrates the ai TAP group from its legacy hook-based approach to the standard test/infra/ Unified CI Infrastructure pattern. Changes: - Added generic hook system to infrastructure scripts: * ensure-infras.bash: Executes group-specific setup-infras.bash hook * run-tests-isolated.bash: Executes group-specific pre-cleanup.bash hook - Migrated ai group infrastructure: * Created infras.lst with infra-mysql84 and docker-pgsql16-single * Renamed seed files: mysql-seed.sql -> seed-mysql.sql, pgsql-seed.sql -> seed-pgsql.sql * Deleted legacy docker-compose.yml, docker-compose-init.bash, docker-compose-destroy.bash * Deleted legacy hooks: pre-proxysql.bash, post-proxysql.bash - Added group-specific MCP configuration: * setup-infras.bash: Configures MCP targets and seeds test data * pre-cleanup.bash: Removes MCP configuration after tests * mcp-config.sql: SQL template for MCP setup * cleanup.sql: SQL template for MCP cleanup * seed-mysql.sql & seed-pgsql.sql: Test data for AI tests - Updated documentation in README.md with usage examples Architecture: - ai is the supergroup (defines infrastructure and shared config) - ai-g1, ai-g2, etc. are subgroups (define test sets in groups.json) - Group-specific hooks live in test/tap/groups/ai/ directory - Infrastructure scripts remain generic and delegate to group hooks	1 month ago
Rene Cannao	2fcfd648e5	Merge remote-tracking branch 'origin/v3.0' into copilot/support-zstd-compression # Conflicts: # test/infra/infra-mysql84/docker-compose-init.bash # test/tap/groups/groups.json	1 month ago
Rene Cannao	8c6a6444db	fix: support zstd compression in mysql84 TAP	1 month ago
Rene Cannao	00b87013cc	Migrate and localize proxysql-tester.py logic - Migrate test-scripts from jenkins-build-scripts to test/scripts/. - Refactor proxysql-tester.py to skip tests missing from groups.json instead of failing. - Ensure correct path resolution and dependency links in localized environment. - Disable automatic pre-hook execution in the Python tester.	1 month ago
Rene Cannao	b7fdec15cd	Fix COM_BINLOG_DUMP fast forward test and isolated environment dependency paths This commit resolves multiple issues in the new CI infrastructure related to binary dependency discovery and COM_BINLOG_DUMP protocol testing. Key changes: - In test_com_binlog_dump_enables_fast_forward-t.cpp: - Fixed a path construction bug where an empty TEST_DEPS resulted in attempting to execute '/mysqlbinlog' (root directory) instead of searching the system PATH. - Added comprehensive diagnostics (diag()) to show connection details, TEST_DEPS status, and detailed file system checks (stat() and 'which') to ensure mysqlbinlog is found and executable. - Improved error reporting by dumping current PATH and CWD on failure. - In infra/control/env-isolated.bash and run-tests-isolated.bash: - Standardized TEST_DEPS and TEST_DEPS_PATH to use workspace-relative paths ('/test-scripts/deps') instead of Jenkins-specific hardcoded paths. - Updated symlink creation logic in run-tests-isolated.bash to correctly map detected binaries (mysqlbinlog, test_binlog_reader-t) into the workspace-relative dependency directory inside the test container.	2 months ago
Rene Cannao	c7ce6b5b92	Eliminate global infrastructure defaults and implement context-aware host derivation - Refactor env-isolated.bash to remove hardcoded mysql84/pgsql16 fallbacks. - Update run-tests-isolated.bash to automatically derive DEFAULT_MYSQL_INFRA and DEFAULT_PGSQL_INFRA from infras.lst. - Create test/tap/groups/legacy/env.sh to explicitly map targets for the legacy group. - Ensure context-correct host resolution for all TAP tests.	2 months ago
Rene Cannao	a4daa4e111	Fix find command noise by pruning restricted directories - Update run-tests-isolated.bash to exclude ci_infra_logs and .git from binary discovery. - Prevents 'Permission denied' errors when find attempts to traverse database data directories.	2 months ago
Rene Cannao	c98f544db5	Enhance infrastructure stability and dynamic configuration - Implement clean slate logic in docker-compose-init.bash (stop containers + wipe data dirs). - Dynamically patch Orchestrator configurations with INFRA_ID-derived ROOT_PASSWORD. - Ensure all literal placeholders are preserved across all infrastructures. - Fix permission issues for PostgreSQL and MySQL log directories. - Verified successful infra-mysql57 initialization and ProxySQL registration.	2 months ago
Rene Cannao	ae646fb425	Finalize test orchestration refactoring and improve error messaging - Refactor run-tests-isolated.bash to provide explicit error messages for missing infras. - Reference infras.lst in error output to clarify setup requirements. - Standardize README.md with updated orchestration roles and manual guide. - Ensure all literal placeholders are preserved in docker-compose.yml files.	2 months ago
Rene Cannao	d289ed44a5	Decouple test execution from infrastructure management - Refactor run-tests-isolated.bash to strictly verify infra state before execution. - Implement infras.lst requirement checking for TAP test groups. - Update test/infra/README.md with clear separation of manual setup, verification, and teardown steps. - Add infras.lst to pgsql17-repl and other discovered test groups. - Ensure stop-proxysql-isolated.bash does not terminate the test runner container.	2 months ago
Rene Cannao	aa9cc7481c	Finalize infra migration and security hardening - Migrate and standardize infra-pgsql17-repl to test/infra/. - Secure all bash scripts with set -e and set -o pipefail for immediate error abort. - Implement reliable INFRA_ID preservation and propagation across all init scripts. - Restore all docker-compose.yml literals to prevent premature shell expansion. - Refactor PostgreSQL wait and post-scripts to use docker exec within isolated network. - Prevent stop-proxysql-isolated.bash from accidentally killing the test-runner container. - Update README.md with detailed instructions for all backend types and clusters. - Decouple pgsql17-repl test group from its internal infrastructure management.	2 months ago
Rene Cannao	1c62557ff2	Standardize migrated infrastructures and fix naming conventions - Restore original directory names (infra-, docker-) to ensure DNS resolution compatibility. - Implement robust relative path derivation for WORKSPACE and REPO_ROOT. - Add SUDO and diag helpers for containerized environment portability. - Mount Docker socket and binary in test-runner for DooD support. - Redirect legacy script calls in test hooks to local infra control paths. - Remove obsolete test/cluster/check_all_nodes.bash.	2 months ago

40 Commits (dca01d4eaf32c8363ff14fb4f74a46deb73e11c3)