161 Commits (3f44229e2835516c77700b5cc84dc9dd460643dd)

Author	SHA1	Message	Date
Rene Cannao	3f44229e28	feat: Add MCP AI Tool Handler for NL2SQL with test script ... Phase 5: MCP Tool Implementation for NL2SQL This commit implements the AI Tool Handler for the MCP (Model Context Protocol) server, exposing NL2SQL functionality as an MCP tool. **New Files:** - include/AI_Tool_Handler.h: Header for AI_Tool_Handler class - Provides ai_nl2sql_convert tool via MCP protocol - Wraps NL2SQL_Converter and Anomaly_Detector - Inherits from MCP_Tool_Handler base class - lib/AI_Tool_Handler.cpp: Implementation - Implements ai_nl2sql_convert tool execution - Accepts parameters: natural_language (required), schema, context_tables, max_latency_ms, allow_cache - Returns JSON response with sql_query, confidence, explanation, cached, cache_id - scripts/mcp/test_nl2sql_tools.sh: Test script for NL2SQL MCP tool - Tests ai_nl2sql_convert via JSON-RPC over HTTPS - 10 test cases covering SELECT, WHERE, JOIN, aggregation, etc. - Includes error handling test for empty queries - Supports --verbose, --quiet options **Modified Files:** - include/MCP_Thread.h: Add AI_Tool_Handler forward declaration and pointer - lib/Makefile: Add AI_Tool_Handler.oo to _OBJ_CXX list - lib/ProxySQL_MCP_Server.cpp: Initialize and register AI tool handler - Creates AI_Tool_Handler with GloAI components - Registers /mcp/ai endpoint - Adds cleanup in destructor **MCP Tool Details:** - Endpoint: /mcp/ai - Tool: ai_nl2sql_convert - Parameters: - natural_language (string, required): Natural language query - schema (string, optional): Database schema name - context_tables (string, optional): Comma-separated table list - max_latency_ms (integer, optional): Max acceptable latency - allow_cache (boolean, optional): Check semantic cache (default: true) **Testing:** Run the test script with: ./scripts/mcp/test_nl2sql_tools.sh [--verbose] [--quiet] See scripts/mcp/test_nl2sql_tools.sh --help for usage. Related: Phase 1-4 (Documentation, Unit Tests, Integration Tests, E2E Tests) Related: Phase 6-8 (User Docs, Developer Docs, Test Docs)	1 month ago
Rene Cannao	147a059781	feat: Add NL2SQL converter with hybrid LLM support ... - Add NL2SQL_Converter with prompt building and model selection - Add LLM clients for Ollama, OpenAI, Anthropic APIs - Update Makefile for new source files	1 month ago
René Cannaò	313f637cf0	Merge branch 'v3.1-vec' into v3.1-MCP1 ... Signed-off-by: René Cannaò <rene.cannao@gmail.com>	1 month ago
Rene Cannao	c86a048d9c	Implement MCP multi-endpoint architecture with dedicated tool handlers ... This commit implements Option 1 (Multiple Tool Handlers) for the MCP module, where each of the 5 endpoints has its own dedicated tool handler with specific tools. ## Architecture Changes - Created MCP_Tool_Handler base class interface for all tool handlers - Each endpoint now has its own dedicated tool handler: - /mcp/config → Config_Tool_Handler (configuration management) - /mcp/query → Query_Tool_Handler (database exploration) - /mcp/admin → Admin_Tool_Handler (administrative operations) - /mcp/cache → Cache_Tool_Handler (cache management) - /mcp/observe → Observe_Tool_Handler (monitoring & metrics) ## New Files Base Interface: - include/MCP_Tool_Handler.h - Base class for all tool handlers Tool Handlers: - include/Config_Tool_Handler.h, lib/Config_Tool_Handler.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp - include/Admin_Tool_Handler.h, lib/Admin_Tool_Handler.cpp - include/Cache_Tool_Handler.h, lib/Cache_Tool_Handler.cpp - include/Observe_Tool_Handler.h, lib/Observe_Tool_Handler.cpp Documentation: - doc/MCP/Architecture.md - Comprehensive architecture documentation ## Modified Files - include/MCP_Thread.h, lib/MCP_Thread.cpp - Added 5 tool handler pointers - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - Use tool_handler base class - lib/ProxySQL_MCP_Server.cpp - Create and pass handlers to endpoints - lib/Makefile - Added new source files ## Implementation Status - Config_Tool_Handler: Functional (get_config, set_config, list_variables, get_status) - Query_Tool_Handler: Functional (wraps MySQL_Tool_Handler, all 18 tools) - Admin_Tool_Handler: Stub implementations (TODO: implement) - Cache_Tool_Handler: Stub implementations (TODO: implement) - Observe_Tool_Handler: Stub implementations (TODO: implement) See GitHub Issue #8 for detailed TODO list. Co-authored-by: Claude <claude@anthropic.com>	1 month ago
Rene Cannao	221ff23991	Add MySQL exploration MCP tools with SQLite catalog ... Implemented MCP (Model Context Protocol) server providing tools for LLM-based MySQL database exploration: - MySQL_Catalog: SQLite-based catalog for LLM external memory with upsert, get, search, list, merge, delete operations and FTS support - MySQL_Tool_Handler: 17+ database exploration tools with guardrails: * Inventory: list_schemas, list_tables * Structure: describe_table, get_constraints, describe_view * Profiling: table_profile, column_profile * Sampling: sample_rows (max 20), sample_distinct (max 50) * Query: run_sql_readonly (max 200 rows, 2s timeout, SELECT-only) * Relationship: suggest_joins, find_reference_candidates * Catalog: catalog_upsert, catalog_get, catalog_search, catalog_list, catalog_merge, catalog_delete - MCP Module Integration: * Added 6 new configuration variables for MySQL tool handler (mysql_hosts, mysql_ports, mysql_user, mysql_password, mysql_schema, catalog_path) * Added MySQL_Tool_Handler pointer to MCP_Threads_Handler * Implemented tool routing in MCP endpoint for tools/list, tools/describe, and tools/call methods - TAP Tests: Updated to expect 14 MCP variables (was 8) Files: - include/MySQL_Catalog.h, lib/MySQL_Catalog.cpp - include/MySQL_Tool_Handler.h, lib/MySQL_Tool_Handler.cpp - include/MCP_Thread.h, lib/MCP_Thread.cpp - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - lib/Makefile, test/tap/tests/mcp_module-t.cpp	1 month ago
Rene Cannao	87fff9e046	Add MCP (Model Context Protocol) module skeleton ... Add new MCP module supporting multiple MCP server endpoints over HTTPS with JSON-RPC 2.0 protocol skeleton. Each endpoint (/mcp/config, /mcp/observe, /mcp/query, /mcp/admin, /mcp/cache) is a distinct MCP server with its own authentication configuration. Features: - HTTPS server using existing ProxySQL TLS certificates - JSON-RPC 2.0 skeleton implementation (actual protocol TBD) - 5 MCP endpoints with per-endpoint auth configuration - LOAD/SAVE MCP VARIABLES admin commands - Configuration file support (mcp_variables section) Implementation follows GenAI module pattern: - MCP_Threads_Handler: Main module handler with variable management - ProxySQL_MCP_Server: HTTPS server wrapper using libhttpserver - MCP_JSONRPC_Resource: Base endpoint class with JSON-RPC skeleton	1 month ago
Rene Cannao	c476f56f97	Add initial GenAI module placeholder ... Implement a new GenAI module for ProxySQL with basic infrastructure: - GenAI_Threads_Handler class for managing GenAI module configuration - Support for genai- prefixed variables in global_variables table - Dummy variables: genai-var1 (string) and genai-var2 (integer) - Config file support via genai_variables section - Flush functions for runtime_to_database and database_to_runtime - Module lifecycle: initialization at startup, graceful shutdown - LOAD/SAVE GENAI VARIABLES admin command infrastructure Core functionality verified: - Config file loading works - Variables persist in global_variables table - Disk save/load via SQL works - Module initializes and shuts down properly Related files: - include/GenAI_Thread.h: New GenAI thread handler class - lib/GenAI_Thread.cpp: Implementation with dummy variables - lib/Admin_Handler.cpp: Added GENAI command vectors and handlers - lib/Admin_FlushVariables.cpp: Added genai flush functions - lib/ProxySQL_Admin.cpp: Added init_genai_variables() and load_save_disk_commands entry - include/proxysql_admin.h: Added function declarations - lib/Makefile: Added GenAI_Thread.oo to build - src/main.cpp: Added module initialization and cleanup - src/proxysql.cfg: Added genai_variables configuration section	1 month ago
Rene Cannao	01d654692d	Integrate sqlite-rembed for text embedding generation ... Add support for sqlite-rembed Rust SQLite extension to enable text embedding generation from remote AI APIs (OpenAI, Nomic, Ollama, Cohere, etc.) within ProxySQL's SQLite3 Server. Changes: 1. Build system integration for Rust static library compilation - Rust toolchain detection in deps/Makefile - Static library target: sqlite3/libsqlite_rembed.a - Linking integration in lib/Makefile and src/Makefile 2. Extension auto-registration in Admin_Bootstrap.cpp - Declare sqlite3_rembed_init() extern C function - Register via sqlite3_auto_extension() after sqlite-vec 3. Documentation updates - doc/sqlite-rembed-integration.md: comprehensive integration guide - doc/SQLite3-Server.md: usage examples and provider list 4. Source code inclusion - deps/sqlite3/sqlite-rembed-source/: upstream sqlite-rembed v0.0.1-alpha.9 The integration follows the same pattern as sqlite-vec (static linking with auto-registration). Provides rembed() function and temp.rembed_clients virtual table for embedding generation. Build requires Rust toolchain (cargo, rustc) and clang/libclang-dev.	2 months ago
Rene Cannao	fbd0d9732b	Add sqlite-vec static extension for vector search in ProxySQL ... This commit integrates sqlite-vec (https://github.com/asg017/sqlite-vec) as a statically linked extension, enabling vector search capabilities in all ProxySQL SQLite databases (admin, stats, config, monitor). Changes: 1. Added sqlite-vec source files to deps/sqlite3/sqlite-vec-source/ - sqlite-vec.c: main extension source - sqlite-vec.h: header for static linking - sqlite-vec.h.tmpl: template header 2. Modified deps/Makefile: - Added target sqlite3/sqlite3/vec.o that copies sources and compiles with flags -DSQLITE_CORE -DSQLITE_VEC_STATIC - Made sqlite3 target depend on vec.o 3. Modified lib/Makefile: - Added $(SQLITE3_LDIR)/vec.o to libproxysql.a prerequisites - Included vec.o in the static library archive 4. Modified lib/Admin_Bootstrap.cpp: - Added extern "C" declaration for sqlite3_vec_init - Enabled load extension support for all databases: - admindb, statsdb, configdb, monitordb, statsdb_disk - Registered sqlite3_vec_init as auto-extension at database open (replacing commented sqlite3_json_init) 5. Updated top-level Makefile: - Made GIT_VERSION fallback to git describe --always when tags missing Result: - Vector search functions (vec0 virtual tables, vector operations) are available in all ProxySQL SQLite databases without runtime dependencies - No separate shared library required; fully embedded in proxysql binary - Extension automatically loaded at database initialization	2 months ago
Rahim Kanji	42864e8867	Improved Tokenizer for PostgreSQL ... - Added `process_pg_typecast()` to handle PostgreSQL type cast syntax (::) - Recognizes type casts in various contexts: 'value'::type, column::type, etc. - Added `process_array_literal()` for PostgreSQL array processing - Handles both ARRAY[] constructor and {} literal syntax - Processes multi-dimensional arrays and nested array structures - Added `process_literal_prefix_type()` for PostgreSQL prefixed literals - Processes E'' escape string constants with backslash escapes - Handles U&'' Unicode string literals with optional UESCAPE clauses - Supports x'' hex string literals and b'' bit string literals - Manages B'' bit strings and bytea literals (\\xDEADBEEF format) - Added `process_replace_boolean()` for boolean literal replacement - Replaces TRUE and FALSE literals with parameter placeholders - Maintains case-insensitive matching (true, True, TRUE, etc.) - Preserves boolean context in expressions and WHERE clauses	2 months ago
Miro Stauder	d020f577b2	reduce duplication in makefiles, use include for common stuff	4 months ago
Miro Stauder	a0f2a9b211	update clickhouse-cpp to v2.6.0	4 months ago
Rahim Kanji	d3592020ac	* Created a separate file for Extended Query messages to reduce clutter in PgSQL_Protocol. ... * Moved get_uint32be() and get_uint16be() to gen_utils.h * Split PgSQL_*_Message into distinct Message and Data components.	8 months ago
Rahim Kanji	24fecc1f6e	Add PostgreSQL extended query (prepared statement) support in ProxySQL ... This commit lays the groundwork for handling PostgreSQL’s extended query protocol (prepared statements) by introducing parsing, it's caching and execution framework stubs. - Implement Parse handling - Implement caching of parse (reuse) - Introduce Execute framework stub for future binding and execution logic.	8 months ago
Rahim Kanji	d0952275ad	Added PgSQL_ExplicitTxnStateMgr in MakeFile	10 months ago
Rahim Kanji	f9232ae50d	Merge remote-tracking branch 'Master/v3.0' into v3.0_session_parameter_tracking	12 months ago
Rahim Kanji	a5b7063cf3	Add validation for PostgreSQL patameters/variables and default PostgreSQL variables ... * Added validation checks when setting PostgreSQL variables to prevent invalid configurations. * Applied the same validation to default PostgreSQL variables to maintain consistency.	12 months ago
Rahim Kanji	35b2dd593d	Separate implementation of SET statement parser for MySQL and PostgreSQL	1 year ago
Yashwant Sahu	004665c6b0	Moved checking of openssl version and finding of openssl library to common place and including common make file.	1 year ago
Yashwant Sahu	3af2af904a	1. Fixed pkg-config cflag, as it was failing with older pkg-config. ... 2. Added a custom path for OpenSSL. Tested with 3.5.0.	1 year ago
Yashwant Sahu	e9b57ac6ef	Using pkg-config to find out insrtalled pkg-config	1 year ago
René Cannaò	2c80f4a935	Merge pull request #4762 from sysown/v3.0_dynamic_fast_forward ... 'COPY ... FROM STDIN' Support with Dynamic Fast Forward and Improved Error Handling - v3.0	1 year ago
Miro Stauder	9e8deb5578	fix test and cleanup	1 year ago
Miro Stauder	456ce7b341	make OpenSSL special case only for almalinux8	1 year ago
Rahim Kanji	7e20a59b0a	Reordered OpenSSL library and include paths to the end to ensure that bundled libraries are prioritized during the build process.	1 year ago
René Cannaò	8218c62dbd	Merge pull request #4740 from sysown/ssl_dynamic_lib ... Using system OpenSSL library	1 year ago
Yashwant Sahu	8cb4f10460	Fixed AlmaLinux builid issues for openssl	1 year ago
Yashwant Sahu	793aba0dd7	Dynamic openssl changes for tap tests.	1 year ago
Rahim Kanji	720441af20	Refactored and Optimized Query Cache with PgSQL Support ... * Added Query Cache Support for PgSQL: * Introduced new variables for PgSQL Query Cache management: * pgsql-query_cache_size_MB * pgsql-query_cache_soft_ttl_pct * pgsql-query_cache_stores_empty_result * pgsql-query_cache_handle_warnings (not yet utilized) * Overwriting Transaction Status: * Sending current transaction state to the client when a result is returned from the query cache. * This currently does not differentiate between normal and error transaction states—both are treated as active transaction states. * Cache Lifetime Management: Replaced manual reference counting with std::shared_ptr to efficiently manage the lifecycle of cache entries, ensuring proper memory deallocation and reducing complexity. * Unified Purging Logic: Refactored the query cache purging logic to use a single thread for both MySQL and PgSQL cache entries. * Code Cleanup: Removed unnecessary methods and data members to streamline the codebase and improve maintainability. * Performance Optimization: Improved memory management, reducing memory footprint and enhancing performance.	1 year ago
Javier Jaramago Fernández	705f50e0f9	Add initial POC for PostgreSQL monitoring support	1 year ago
René Cannaò	d4bd93672e	Merge pull request #79 from sysown/v2.x_refactore_query_processor ... Refactored Query Processor [WIP]	1 year ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture ... - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	1 year ago
René Cannaò	cbf4628ea3	Merge remote-tracking branch 'proxysql27/v2.7' into v2.x_postgres_merge27_20240917	1 year ago
Rahim Kanji	745449fcbc	Removed Client_Session.h and Client_Session.cpp	1 year ago
Rene Cannao	d85430b709	Implementation of PROXY protocol V1 ... This commit introduces: - class ProxyProtocolInfo() , that: - performs parsing - validates subnet - run automated tests in DEBUG build - variable mysql-proxy_protocol_networks . Accepted values: - empty string: disables PROXY protocol - '*' : allows connections from any IP - comma separated list of subnets - automated testing in DEBUG build during start - export of PROXY protocol information in internal session, using PROXY_V1 - a TAP test to verify various connections	2 years ago
Rene Cannao	0a391bd587	Some code cleanup	2 years ago
Rene Cannao	a593f215d1	Creating BaseHGC (Base HostGroup Container)	2 years ago
Rene Cannao	cf61603375	Creating BaseSrvList ... Template for MySrvList and PgSQL_SrvList . This is an initial commit to start merging the two Hostgoup Managers	2 years ago
Rene Cannao	a1fd4af1a6	Splitting ProxySQL_Admin	2 years ago
Rene Cannao	465eaef87e	Merge branch 'v2.x_postgres' into v2.x_240630	2 years ago
Rahim Kanji	65fc633cf5	Generic error handling, transaction detection and handling, bug fixes ... * Implemented generic error handling to cover all the scenarios. * Extraction of results from PGconn in case of connection errors. Introduced a patch to facilitate this. * Added transaction handling. * Ready packet now includes transaction status. * Split command completion and ready packet to increase flexibility. * Resolved multiple bugs to enhance stability.	2 years ago
Rahim Kanji	06a32365cc	Merge remote-tracking branch 'Master_POSTGRES/v2.x_postgres' into v2.x_postgres_backend	2 years ago
Rahim Kanji	dcbe140199	Route query to PostgreSQL backend server	2 years ago
Miro Stauder	da61f23eb3	Merge branch 'v2.x' into v2.6.x-freebsd_fixes	2 years ago
René Cannaò	0cfab2616f	First attempt to use Base_Thread and Base_Session	2 years ago
René Cannaò	38c6c2ac7b	Merge branch 'v2.x-main' into v2.x_postgres_240422 ... Merged main ProxySQL code up to commit 959c629	2 years ago
René Cannaò	7a50a68f28	Split of ProxySQL_Admin.cpp ... ProxySQL_Admin.cpp as become very big, thus it is time to split it in smaller files for better reasibility and processing. Still a work in progress.	2 years ago
René Cannaò	dad6f0f344	Split of MySQL_Protocol.cpp into multiple files	2 years ago
René Cannaò	e7aa5ff4c5	Split of MySQL_HostGroups_Manager.cpp into multiple files ... First commit to split MySQL_HostGroups_Manager.cpp into multiple files. More to split.	2 years ago
René Cannaò	0a2cd03672	Refactoring to split Query_Processor.cpp into multiple files	2 years ago