proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	6 days ago
Rene Cannao	200fdecb8d	Apply AI agent review fixes to PR #5374 Critical Fixes: - Fix NULL pointer dereferences in MySQL_Logger.cpp: * Add null checks for myds->myconn->parent in log_audit_entry() * Fix cleanup code in log_request() and log_audit_entry() * Change sprintf to snprintf for buffer safety - Remove redundant pthread_rwlock_init in debug.cpp: * filters_rwlock already initialized with PTHREAD_RWLOCK_INITIALIZER * Re-initialization caused undefined behavior per POSIX Major Improvements: - Add bounded wait helper wait_for_glo_mth() in proxysql_utils.h: * Replaces indefinite while loops with 10s timeout * Applied to 26 locations across 4 files Build System: - Require C++17 in Makefile (remove C++11 fallback): * Code uses C++17 structured bindings extensively * Cleanest solution vs adding fallback guards to 27+ locations Compilation fixes: - Add unistd.h include for usleep - Remove conflicting GloMyMon extern declaration Files changed: - Makefile - include/proxysql_utils.h - lib/MySQL_Logger.cpp - lib/debug.cpp - lib/MySQL_Monitor.cpp - lib/ProxySQL_Admin.cpp - lib/ClickHouse_Server.cpp - src/SQLite3_Server.cpp	1 week ago
Rene Cannao	3c85c593cf	Add GloMTH initialization wait to all monitor and server threads Fix race condition where threads were accessing GloMTH before it was fully initialized, causing Valgrind errors in get_variable_int() and related functions. Added wait loop pattern (while GloMTH==NULL { usleep(50000); }) followed by usleep(100000) to ensure GloMTH is fully initialized before use: In lib/MySQL_Monitor.cpp: - monitor_connect(), monitor_ping(), monitor_read_only() - monitor_group_replication(), monitor_galera(), monitor_replication_lag() - monitor_AWS_Aurora_thread_HG(), monitor_aws_aurora() - init_mysql_thread_struct() In src/SQLite3_Server.cpp: - SQLite3_Server_session_handler In lib/ProxySQL_Admin.cpp: - admin_session_handler In lib/ClickHouse_Server.cpp: - ClickHouse_Server_session_handler The pattern ensures threads wait for GloMTH initialization during startup, then check for NULL during shutdown to exit cleanly.	1 week ago
Rene Cannao	3e37337877	Use RAII for sqlite3 statements across codebase - Convert prepare_v2() calls to use unique_ptr RAII pattern - Remove manual sqlite3_finalize calls (now handled by RAII) - Update flush_*_variables functions to use new prepare_v2 return pattern - Apply changes to Admin, Cluster, Monitor, Catalog, and other modules	1 week ago
Rene Cannao	ba86be70a6	Fixed some bugs in PR 5358	1 week ago
René Cannaò	02ea4f90be	Merge pull request #5360 from sysown/v3.0-5359 fix: crash on macOS/FreeBSD when running PostgreSQL queries	1 week ago
Rene Cannao	ac71e12a95	refactor: migrate RSA key generation to OpenSSL 3.0 EVP_PKEY API Replace deprecated OpenSSL RSA APIs with OpenSSL 3.0 EVP_PKEY keygen API: - Replace RSA_new() + RSA_generate_key_ex() with EVP_PKEY_keygen() - Replace PEM_write_bio_RSAPrivateKey() with PEM_write_bio_PrivateKey() - Remove EVP_PKEY_assign_RSA() - use EVP_PKEY directly - Update write_rsa_key() to accept EVP_PKEY instead of RSA* This eliminates deprecation warnings for: - RSA_new() - RSA_generate_key_ex() - RSA_free() - EVP_PKEY_assign_RSA() - PEM_write_bio_RSAPrivateKey() Note: MD5_* deprecation warnings in PgSQL_Protocol.cpp remain as a separate issue (scram-md5 authentication uses legacy MD5 API).	2 weeks ago
René Cannaò	c45fabdb6d	fix: correct macOS executable SHA1 implementation - Use strlen(buff) for actual path length instead of bufsize (_NSGetExecutablePath does not modify bufsize on success) - Add explicit null-termination after strncpy for safety - Pass sizeof(buff) instead of sizeof(buff)-1 to _NSGetExecutablePath Addresses review feedback from gemini-code-assist.	2 weeks ago
René Cannaò	e3bdd3c07d	main: add macOS support for executable SHA1 calculation Use _NSGetExecutablePath() on macOS instead of /proc/self/exe which doesn't exist on macOS. Also use realpath() to resolve symlinks and get the canonical path, consistent with behavior on Linux/FreeBSD.	2 weeks ago
René Cannaò	a360dc22ae	Merge pull request #5308 from sysown/v3.0_mac Support for ProxySQL compilation on macOS (Darwin/Apple Silicon)	2 weeks ago
Rene Cannao	1af768f932	Fix critical issues in GenAI code (PR #5339 ) This commit addresses critical issues identified in the PR review: 1. Build system redundancy (src/Makefile): - Remove redundant PROXYSQLGENAI conditionals in linking commands - The flag doesn't affect linking, so simplify to PROXYSQLCLICKHOUSE only 2. Missing PROXYSQLGENAI guard (lib/Admin_Handler.cpp): - Add #ifdef PROXYSQLGENAI around MCP VARIABLES DISK commands - Ensures MCP commands are only available when GenAI is enabled 3. Broken retry logic (lib/LLM_Clients.cpp): - Remove misleading is_retryable_error() checks that used stale values - last_curl_code and last_http_code were never updated - Simplify to retry on empty responses only (documented limitation) - Fix thread-safety: use thread_local std::mt19937 instead of rand() 4. Resource leak (lib/MySQL_Tool_Handler.cpp): - Clean up previously created connections on init failure - Both mysql_init and mysql_real_connect error paths now clean up 5. Race condition (lib/Query_Tool_Handler.cpp): - Add missing pool_lock in find_connection() - Prevents race condition when accessing connection_pool Fixes identified by automated PR review agents.	3 weeks ago
Rene Cannao	48bc7dd7bf	Merge v4.0 GenAI features into v3.0 with conditional compilation This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)	3 weeks ago
Javier Jaramago Fernández	1a7668b73d	fix: SQLite3 server support for MySQL client '8.1.0' For more details check commit #ecf4f69.	4 weeks ago
Rene Cannao	87d8c25074	Reapply "fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins'" This reverts commit `48260cbf09`.	4 weeks ago
Rene Cannao	48260cbf09	Revert "fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins'" This reverts commit `df221617c0`.	1 month ago
Javier Jaramago Fernández	df221617c0	fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins' Since 'proxy_sqlite3_open_v2' is now being set by default due to the new dedicated TU ('proxy_sqlite3_symbols.cpp'). 'LoadPlugins' might not be called, not properly setting 'SQLITE_CONFIG_URI' and invalidating many configuration that are currently specified in the conn URI.	1 month ago
Rene Cannao	03e58146ea	fix: Re-enable SQLite3DB::LoadPlugin() with allow_load_plugin flag This addresses an issue from PR #22 where LoadPlugin() was completely disabled. The function performs necessary initialization even when no plugin is loaded (initializes built-in sqlite3 function pointers). Changes: - Added `const bool allow_load_plugin = false` flag in LoadPlugin() - Modified `if (plugin_name)` to `if (plugin_name && allow_load_plugin == true)` - Re-enabled the LoadPlugin() call in LoadPlugins() The plugin loading code remains disabled (allow_load_plugin=false) while the function pointer initialization from built-in SQLite3 now works correctly. TODO: Revisit plugin loading safety mechanism to allow actual plugin loading.	1 month ago
Rene Cannao	b4f521c634	Merge v3.1-MCP2 into v3.1-vec Conflict resolution summary: Features preserved from v3.1-vec: - mcp_use_ssl variable (HTTP/HTTPS mode support) - RAG tool handler support - FTS (Full Text Search) functionality in MySQL_Tool_Handler Changes accepted from v3.1-MCP2: - Schema isolation for catalog (removed mcp_catalog_path/fts_path variables) - Schema parameter added to catalog functions - Comprehensive query rewriting improvements in Query_Tool_Handler - Format string fixes in vector_db_performance-t.cpp Key decisions: - catalog_path/fts_path: Removed per v3.1-MCP2 commit `35b0b224` (intentional) - MySQL_Catalog.cpp: Accepted v3.1-MCP2 version (schema isolation improvements) - Query_Tool_Handler.cpp: Accepted v3.1-MCP2 version (significant improvements) - Makefile: Kept v3.1-vec's EXCLUDE_TRACKING_VARIABLES (correct spelling) Build notes: - MySQL_Tool_Handler now requires schema parameter for catalog operations - Catalog path is hardcoded to datadir/mcp_catalog.db - FTS functionality preserved but may need updates for schema isolation	1 month ago
Rene Cannao	6ce0538489	Keep main.cpp only; remove accidental backup from commits	1 month ago
Rene Cannao	f877366a68	Restore commented SQLite3DB::LoadPlugin reference with TODO	1 month ago
Rene Cannao	4f0e6e0a4e	Disable sqlite3 plugin function replacement; warn instead	1 month ago
Rene Cannao	2dfd61a958	Replace remaining direct sqlite3_* calls with proxy_sqlite3_* equivalents (address code-review)	1 month ago
René Cannaò	d43ae6e121	Surgical fixes for macOS compatibility: headers, types, and Makefile linking	1 month ago
Rene Cannao	8dc4246bdc	Introduce canonical proxy_sqlite3 symbol TU; update lib Makefile; remove MAIN_PROXY_SQLITE3 from main.cpp	1 month ago
Rahim Kanji	5b8bb1952e	Merge remote-tracking branch 'wqv3.1-vec' into v3.1_mcp-http-ssl-toggle	1 month ago
Rahim Kanji	f7397f633c	Fix catalog search to use FTS5 and enhance test suite The catalog_fts FTS5 virtual table was being created but the search() function was using slow LIKE queries instead of FTS5 MATCH operator. Changes to lib/MySQL_Catalog.cpp: - Use FTS5 MATCH with INNER JOIN to catalog_fts when query provided - Add BM25 relevance ranking (ORDER BY bm25(f) ASC) - Significant performance improvement: O(log n) vs O(n) Changes to scripts/mcp/test_catalog.sh: - Add 8 new FTS5-specific tests (CAT013-CAT020): - Multi-term search (AND logic) - Phrase search with quotes - Boolean operators (OR, NOT) - Prefix search with wildcards - Kind and tags filter combinations - Relevance ranking verification - Add SSL/HTTP support with auto-detection - New options: --ssl, --no-ssl, MCP_USE_SSL env var - Fix endpoint path: /query -> /mcp/querywq	1 month ago
Rahim Kanji	ddc4e65706	Add plain HTTP support for MCP server and fix SSL/port restart issues * Add full support for both HTTP and HTTPS modes in MCP server via the mcp_use_ssl configuration variable, enabling plain HTTP for development and HTTPS for production with proper certificate validation * Server now automatically restarts when SSL mode or port configuration changes, fixing silent configuration failures where changes appeared to succeed but didn't take effect until manual restart. Features: - Explicit support for HTTP mode (mcp_use_ssl=false) without SSL certificates - Explicit support for HTTPS mode (mcp_use_ssl=true) with certificate validation - Configurable via configure_mcp.sh with --no-ssl or --use-ssl flags - Settable via admin interface: SET mcp-use_ssl=true/false - Automatic restart detection for SSL mode changes (HTTP ↔ HTTPS) - Automatic restart detection for port changes (mcp_port)	1 month ago
Rene Cannao	d9346fe64d	feat: Add AI features manager foundation - Add AI_Features_Manager coordinator class - Add AI_Vector_Storage interface (stub) - Add Anomaly_Detector class (stub for Phase 3) - Update includes and main initialization	1 month ago
Rene Cannao	6d83ff1680	Fix: unwrap ProxySQL response format in MCP tools and fix config syntax - Unwrap ProxySQL's {"success": ..., "result": ...} wrapper in tool responses for MCP protocol compliance - Fix proxysql.cfg missing closing brace for mcp_variables section	1 month ago
René Cannaò	d504c93b45	Fix formatting in proxysql.cfg Signed-off-by: René Cannaò <rene.cannao@gmail.com>	1 month ago
Rene Cannao	119ca5003a	Fix compilation errors in debug build - Move explicit template instantiations for send_ok_msg_to_client and send_error_msg_to_client to after template definitions - Add missing closing brace for init_mcp_variables() - Fix missing #endif and closing brace for GloMCPH shutdown block	1 month ago
René Cannaò	313f637cf0	Merge branch 'v3.1-vec' into v3.1-MCP1 Signed-off-by: René Cannaò <rene.cannao@gmail.com>	1 month ago
Rene Cannao	245e61ee86	Make MCP_Threads_Handler a standalone independent class Remove unnecessary inheritance from MySQL_Threads_Handler. The MCP module should be independent and not depend on MySQL/PostgreSQL thread handlers. Changes: - MCP_Threads_Handler now manages its own pthread_rwlock_t for synchronization - Simplified init() signature (removed unused num/stack parameters) - Added ProxySQL_Main_init_MCP_module() call in main initialization phase - Include only standard C++ headers (pthread.h, cstring, cstdlib)	1 month ago
Rene Cannao	87fff9e046	Add MCP (Model Context Protocol) module skeleton Add new MCP module supporting multiple MCP server endpoints over HTTPS with JSON-RPC 2.0 protocol skeleton. Each endpoint (/mcp/config, /mcp/observe, /mcp/query, /mcp/admin, /mcp/cache) is a distinct MCP server with its own authentication configuration. Features: - HTTPS server using existing ProxySQL TLS certificates - JSON-RPC 2.0 skeleton implementation (actual protocol TBD) - 5 MCP endpoints with per-endpoint auth configuration - LOAD/SAVE MCP VARIABLES admin commands - Configuration file support (mcp_variables section) Implementation follows GenAI module pattern: - MCP_Threads_Handler: Main module handler with variable management - ProxySQL_MCP_Server: HTTPS server wrapper using libhttpserver - MCP_JSONRPC_Resource: Base endpoint class with JSON-RPC skeleton	1 month ago
Rene Cannao	c476f56f97	Add initial GenAI module placeholder Implement a new GenAI module for ProxySQL with basic infrastructure: - GenAI_Threads_Handler class for managing GenAI module configuration - Support for genai- prefixed variables in global_variables table - Dummy variables: genai-var1 (string) and genai-var2 (integer) - Config file support via genai_variables section - Flush functions for runtime_to_database and database_to_runtime - Module lifecycle: initialization at startup, graceful shutdown - LOAD/SAVE GENAI VARIABLES admin command infrastructure Core functionality verified: - Config file loading works - Variables persist in global_variables table - Disk save/load via SQL works - Module initializes and shuts down properly Related files: - include/GenAI_Thread.h: New GenAI thread handler class - lib/GenAI_Thread.cpp: Implementation with dummy variables - lib/Admin_Handler.cpp: Added GENAI command vectors and handlers - lib/Admin_FlushVariables.cpp: Added genai flush functions - lib/ProxySQL_Admin.cpp: Added init_genai_variables() and load_save_disk_commands entry - include/proxysql_admin.h: Added function declarations - lib/Makefile: Added GenAI_Thread.oo to build - src/main.cpp: Added module initialization and cleanup - src/proxysql.cfg: Added genai_variables configuration section	2 months ago
Rene Cannao	01d654692d	Integrate sqlite-rembed for text embedding generation Add support for sqlite-rembed Rust SQLite extension to enable text embedding generation from remote AI APIs (OpenAI, Nomic, Ollama, Cohere, etc.) within ProxySQL's SQLite3 Server. Changes: 1. Build system integration for Rust static library compilation - Rust toolchain detection in deps/Makefile - Static library target: sqlite3/libsqlite_rembed.a - Linking integration in lib/Makefile and src/Makefile 2. Extension auto-registration in Admin_Bootstrap.cpp - Declare sqlite3_rembed_init() extern C function - Register via sqlite3_auto_extension() after sqlite-vec 3. Documentation updates - doc/sqlite-rembed-integration.md: comprehensive integration guide - doc/SQLite3-Server.md: usage examples and provider list 4. Source code inclusion - deps/sqlite3/sqlite-rembed-source/: upstream sqlite-rembed v0.0.1-alpha.9 The integration follows the same pattern as sqlite-vec (static linking with auto-registration). Provides rembed() function and temp.rembed_clients virtual table for embedding generation. Build requires Rust toolchain (cargo, rustc) and clang/libclang-dev.	2 months ago
Rahim Kanji	c0f99c0e15	Refactor: Improved Prepared-Statement Cache Design (Lock-Free Hot Path) #5211 Concurrency and Memory Management * Lock-Free Ref Counting: Replaced global mutex-protected integer reference counts with `std::atomic<uint32_t>` within `PgSQL_STMT_Global_info`, eliminating lock contention during statement referencing. * Modern Ownership: Adopted std::shared_ptr<const PgSQL_STMT_Global_info> for global and local storage, providing automatic, thread-safe memory and lifecycle management. * Memory Optimization: Removed redundant auxiliary maps `global_id_to_stmt_names` and `map_stmt_id_to_info` from local and global statement managers respectively, reducing overall memory overhead. * Optimized Purging: Statement removal logic was simplified for efficiently identifying and cleaning up unused statements. Hot Path Performance (`BIND`, `DESCRIBE`, `EXECUTE`) * Bypassed Global Lookups: Local session maps now store the `shared_ptr` directly, removing the need to acquire the global lock and search the global map during hot path operations. * Direct Refcount Manipulation: Refcount modification functions now operate directly on the passed statement object, eliminating the overhead of searching the global map to find the object pointer based on statement id. Safety and Protocol Logic (`PARSE`) * Efficient Statement Reuse: Implemented a local fast path check for the unnamed statement (`""`), allowing immediate reuse of an identical query (same hash) upon re-parse, which bypasses global processing and locks. Cleanup * Cleaned up and class rename `PgSQL_STMT_Manager_v14` -> `PgSQL_STMT_Manager`.	3 months ago
Miro Stauder	d020f577b2	reduce duplication in makefiles, use include for common stuff	4 months ago
René Cannaò	c5a0a0011f	Merge pull request #5137 from sysown/v3.0_pgsql_query_cancel_terminate_issue_5136 Implement Backend Query Cancellation and Backend Termination for PostgreSQL	4 months ago
Miro Stauder	a0f2a9b211	update clickhouse-cpp to v2.6.0	5 months ago
Miro Stauder	88c34b30a5	fix issues by updating libconfig	5 months ago
Rahim Kanji	9cb605f4fb	Replaced rand() with random()	5 months ago
Rahim Kanji	8d8577b0ad	Ensure PgSQL_Threads_Handler and PgSQL_HostGroups_Manager objects are released when ProxySQL shuts down or restarts.	6 months ago
Rahim Kanji	344ba25777	Add Watchdog PROXYSQLTEST covering MySQL and PgSQL threads	7 months ago
Rahim Kanji	e4c0f5ccd8	Refactored Watchdog and integrated PostgreSQL thread monitoring into its loop.	7 months ago
Rahim Kanji	24fecc1f6e	Add PostgreSQL extended query (prepared statement) support in ProxySQL This commit lays the groundwork for handling PostgreSQL’s extended query protocol (prepared statements) by introducing parsing, it's caching and execution framework stubs. - Implement Parse handling - Implement caching of parse (reuse) - Introduce Execute framework stub for future binding and execution logic.	8 months ago
Javier Jaramago Fernández	811874bad1	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-ff_conns_eof_caps_match	11 months ago
René Cannaò	1123da34bb	Merge pull request #4904 from sysown/v3.0-4877 Fix certificate chain send during SSL handshake - Closes #4877	11 months ago
Javier Jaramago Fernández	7955dd21a8	Fix MySQL 8.4 replication lag monitoring - Closes #4727 Added support for new syntax to simulator.	11 months ago
Javier Jaramago Fernández	f206e1be05	Fix certificate chain send during SSL handshake - Closes #4877 Removed usages of 'SSL_CTX_add_extra_chain_cert'. Manually adding certs to the certificate chain shouldn't be required, using 'SSL_CTX_load_verify_locations' with the right arguments should be enough, allowing OpenSSL to correctly select the certificates to be used for validation purposes.	11 months ago

1 2 3 4 5 ...

570 Commits (v3.0-MCP_multi)