193 Commits (ade0130e67e88b9052d0dc4e76d48d98d3328485)

Author	SHA1	Message	Date
Rene Cannao	9685cdaa4b	mcp discovery: enforce target-scoped run model and add protocol-aware static harvesting (mysql+pgsql) ... Refactor the MCP query/discovery stack to remove prototype-era MySQL-only assumptions and make discovery/catalog semantics explicitly target-scoped. This commit is intentionally not backward compatible with legacy single-target catalog metadata. Key implementation details: - Discovery_Schema: - runs table now stores target_id, protocol, and server_version - resolve_run_id() now requires target_id and resolves schemas within target scope - create_run() now records target/protocol/server version - added legacy schema detection + destructive catalog table rebuild when old runs layout is found - fixed resultset lifetime/escaping issues in target-aware run resolution - New PostgreSQL static harvester: - added PgSQL_Static_Harvester class and build wiring - harvests schemas/objects/columns/indexes/fks/view definitions into Discovery_Schema - maps pg metadata to object_id-based insert APIs used by Discovery_Schema - Query_Tool_Handler: - constructor simplified to catalog_path only (no mcp-mysql_* runtime ctor deps) - discovery.run_static now requires target_id and dispatches protocol-aware harvester at runtime - catalog.*, agent.run_start, and llm.* tool contracts now require target_id when resolving run_id - all run_id resolution call sites switched to resolve_run_id(target_id, ...) - list_schemas catalog query now scoped by target_id via runs join - digest logging path now resolves run_id in target context - ProxySQL_MCP_Server: - updated Query_Tool_Handler construction to new signature - Docs updated: - added explicit target_id scoping guidance for discovery/catalog/agent/llm workflows - clarified protocol-aware routing and noted legacy examples still present in large script docs Build validation: - Recompiled changed objects with PROXYSQLGENAI=1: - Query_Tool_Handler.oo - Discovery_Schema.oo - PgSQL_Static_Harvester.oo - ProxySQL_MCP_Server.oo	5 days ago
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra ... This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	6 days ago
Rene Cannao	4a8b224038	MCP TAP/docs: migrate tests and documentation from legacy MCP mysql vars to profile-based routing ... This commit migrates MCP TAP assets and related docs away from deprecated `mcp-mysql_*` style configuration toward the profile-based model centered on auth profiles, target profiles, and `target_id` routing. Changes included: - Updated TAP configuration helpers and test inputs to build and load MCP auth/target profiles. - Replaced legacy variable-driven setup assumptions with runtime profile loading (`LOAD MCP PROFILES ...`). - Refreshed documentation snippets and examples to describe the profile-based configuration flow. Motivation: - Keep test and documentation surfaces aligned with the current MCP architecture. - Eliminate ambiguity between old POC paths and supported profile-driven routing. Outcome: - TAP and docs now consistently describe and exercise MCP routing through profile tables and `target_id` selection.	6 days ago
Rene Cannao	013864b36f	MCP: introduce profile-based target/auth routing and unified LOAD/SAVE MCP PROFILES commands ... This commit introduces the core MCP routing architecture that decouples client-facing target selection from backend connection details. Architecture introduced: - New MCP auth profile table to store backend credentials/policy (`mcp_auth_profiles`). - New MCP target profile table to map opaque `target_id` -> protocol/hostgroup/auth profile (`mcp_target_profiles`). - Runtime mirrors for both profile tables. Behavioral changes: - MCP query tools resolve execution context by `target_id` and route internally by protocol + hostgroup. - Client no longer needs backend-specific connection details; only logical target identifiers. - Unified admin commands for profile lifecycle management: - `LOAD MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` - `SAVE MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` Design goals: - Support multi-backend MCP execution (MySQL + PostgreSQL) behind one endpoint. - Provide server-side credential management and cleaner operational model. - Replace legacy one-backend POC configuration with scalable target/auth abstractions. Result: - MCP routing is now dynamic, protocol-agnostic at the client layer, and fully managed through profile tables and unified load/save commands.	6 days ago
Rene Cannao	a69b925932	Minor typo fixes	1 week ago
Rene Cannao	d90be4080a	Add documentation for Prometheus protocol labels feature ... Add comprehensive documentation describing the protocol labels feature introduced in this branch to resolve metric collisions between MySQL and PostgreSQL prometheus metrics. Related: #5068, #5069	2 weeks ago
René Cannaò	a360dc22ae	Merge pull request #5308 from sysown/v3.0_mac ... Support for ProxySQL compilation on macOS (Darwin/Apple Silicon)	2 weeks ago
Rene Cannao	b4f521c634	Merge v3.1-MCP2 into v3.1-vec ... Conflict resolution summary: **Features preserved from v3.1-vec:** - mcp_use_ssl variable (HTTP/HTTPS mode support) - RAG tool handler support - FTS (Full Text Search) functionality in MySQL_Tool_Handler **Changes accepted from v3.1-MCP2:** - Schema isolation for catalog (removed mcp_catalog_path/fts_path variables) - Schema parameter added to catalog functions - Comprehensive query rewriting improvements in Query_Tool_Handler - Format string fixes in vector_db_performance-t.cpp **Key decisions:** - catalog_path/fts_path: Removed per v3.1-MCP2 commit 35b0b224 (intentional) - MySQL_Catalog.cpp: Accepted v3.1-MCP2 version (schema isolation improvements) - Query_Tool_Handler.cpp: Accepted v3.1-MCP2 version (significant improvements) - Makefile: Kept v3.1-vec's EXCLUDE_TRACKING_VARIABLES (correct spelling) **Build notes:** - MySQL_Tool_Handler now requires schema parameter for catalog operations - Catalog path is hardcoded to datadir/mcp_catalog.db - FTS functionality preserved but may need updates for schema isolation	1 month ago
Rene Cannao	02918d18b8	Fix PR #25 Review: All AI code reviewer feedback addressed ... This commit addresses all recommendations from CodeRabbit, Gemini Code Assist, and Copilot for PR #25 (FTS security and code quality improvements). Critical Security Fixes: - MCP_Thread.cpp: Rollback fts_path on reset failure to keep config consistent - MySQL_FTS.cpp: Add escape_mysql_identifier() for MySQL query identifier escaping - MySQL_FTS.cpp: Add unique hash-based fallback to sanitize_name() for empty strings - MySQL_FTS.cpp: Add where_clause validation to block dangerous SQL patterns Memory Safety Fixes: - MySQL_FTS.cpp: Fix indexes_result memory leak on early return in search() - MySQL_FTS.h: Delete copy/move operations to prevent accidental resource duplication Thread Safety Documentation: - MySQL_Tool_Handler.cpp: Add comment explaining FTS lock design rationale Test Script Improvements: - test_mcp_fts.sh: Add curl timeouts (5s connect, 30s max) - test_mcp_fts.sh: Remove unused delete_response variable - test_mcp_fts_detailed.sh: Make cleanup tolerant of non-existent indexes Build Fixes: - Makefile: Fix EXCLUDE_TRACKING_VARAIABLES typo to EXCLUDE_TRACKING_VARIABLES - vector_db_performance-t.cpp: Fix printf format specifiers to %lld with cast Schema Fixes: - Query_Tool_Handler.cpp: Change fts_index_table columns schema from string to array Code Cleanup: - MySQL_Tool_Handler.cpp: Remove all remaining debug fprintf statements (34 lines) - Documentation: Change "Full Text" to "Full-Text" (hyphenated) Total: ~50 fixes across 10 files	1 month ago
Rene Cannao	a10c09bcc9	Fix PR #21 review: Security, memory safety, thread safety, and code cleanup ... Security fixes: - Add escape_identifier() helper for proper SQLite identifier escaping - Replace sanitize_name() with allowlist validation (ASCII letters, digits, underscore only) - Fix MATCH clause FTS5 operator injection by wrapping query in double quotes - Apply escape_identifier() to all DDL statements (CREATE, DROP, triggers) Memory safety fixes: - Replace VLA with std::vector in MySQL_FTS::init(), add delete on error path - Fix memory leak: free error string before return in list_indexes() - Fix reindex_json["error"] potential exception using .value() with default Thread safety fixes: - reinit_fts(): Add mutex lock around pointer swap - reset_fts_path(): Move blocking init() outside lock, only swap pointer under lock Code cleanup: - Remove 7 debug fprintf statements from Query_Tool_Handler.cpp - Remove unused #include <memory> from MySQL_FTS.h Test script security fixes: - Use MYSQL_PWD environment variable instead of -p"..." for password - Add escape_sql() function and apply to INSERT statement - Fix CURL_OPTS quoting: ${CURL_OPTS:+"${CURL_OPTS}"} - Remove unused FTS_INDEX_NAME and SEARCH_QUERIES variables Documentation fixes: - Fix bare URL to markdown link format - Add code block language identifiers (text, bash)	1 month ago
René Cannaò	a831670a79	Merge pull request #21 from rahim-kanji/v3.1_fts-support ... Add full-text search (FTS) tools to MCP query server	1 month ago
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec ... Signed-off-by: René Cannaò <rene@proxysql.com>	1 month ago
René Cannaò	62bc15fd88	Improve macOS build system OpenSSL detection and documentation ... - Support OPENSSL_ROOT_DIR environment variable for better Homebrew integration - Fix OpenSSL library path on Darwin (use lib instead of lib64) - Improve OpenSSL version extraction using grep/sed for better compatibility - Add TAP test build instructions to BUILD-MACOS.md	1 month ago
René Cannaò	d10cdd3db7	Updated macOS build instructions in INSTALL.md and added doc/BUILD-MACOS.md	1 month ago
Rahim Kanji	8e2230c3e2	Add FTS_User_Guide.md	1 month ago
Rene Cannao	acd05b60a9	Organize RAG test files properly ... - Move RAG test files to dedicated test/rag/ directory - Create proper Makefile in test/rag/ directory - Remove build_rag_test.sh script (replaced by Makefile) - Revert changes to main test/Makefile - Update documentation to reflect new test structure	1 month ago
Rene Cannao	ed65b6905b	Remove mistakenly created Doxygen files ... - Remove Doxyfile that was added by mistake - Remove .doxygen files from doc/ directory that were generated by mistake - Clean up documentation artifacts that shouldn't have been committed	1 month ago
Rene Cannao	ad166c6b8a	docs: Add comprehensive Doxygen documentation for RAG subsystem ... - Enhanced inline Doxygen comments in RAG_Tool_Handler.h and RAG_Tool_Handler.cpp - Added detailed parameter descriptions, return values, and cross-references - Created Doxyfile for documentation generation - Added documentation summary and guidelines - Documented all RAG tools with their schemas and usage patterns - Added security and performance considerations documentation The RAG subsystem is now fully documented with comprehensive Doxygen comments that provide clear guidance for developers working with the codebase.	1 month ago
Rene Cannao	3daaa5c592	feat: Implement RAG (Retrieval-Augmented Generation) subsystem ... Adds a complete RAG subsystem to ProxySQL with: - RAG_Tool_Handler implementing all MCP tools for retrieval operations - Database schema with FTS and vector support - FTS, vector, and hybrid search capabilities - Fetch and refetch tools for document/chunk retrieval - Admin tools for monitoring - Configuration variables for RAG parameters - Comprehensive documentation and test scripts Implements v0 deliverables from RAG blueprint: - SQLite schema initialization - Source registry management - MCP tools: search_fts, search_vector, search_hybrid, get_chunks, get_docs, fetch_from_source, admin.stats - Unit/integration tests and examples	1 month ago
Rene Cannao	aced263367	docs: Update MCP documentation to reflect current implementation ... This commit completes a comprehensive review and update of all MCP-related documentation: - Remove deprecated FastAPI POC files - Update Architecture.md to reflect implemented multi-endpoint architecture - Update VARIABLES.md with new AI endpoint authentication - Update Tool_Discovery_Guide.md with discovery and LLM tools - Update FTS_Implementation_Plan.md to reflect implemented status - Update Vector_Embeddings_Implementation_Plan.md to show planned status - Update Database_Discovery_Agent.md to clarify conceptual design status - Update scripts/mcp/README.md with current multi-handler architecture - Update STDIO_BRIDGE_README.md with complete tool list All documentation now accurately reflects the current ProxySQL MCP implementation with 6 dedicated tool handlers and two-phase discovery.	1 month ago
Rene Cannao	35b0b224ff	refactor: Remove mcp-catalog_path variable and hardcode catalog path ... Remove the mcp-catalog_path configuration variable and hardcode the catalog database path to datadir/mcp_catalog.db for stability. Rationale: The catalog database is session state, not user configuration. Runtime swapping of the catalog could cause tables to be missed and the catalog to fail even if it was succeeding a second earlier. Changes: - Removed catalog_path from mcp_thread_variables_names array - Removed mcp_catalog_path from MCP_Thread variables struct - Removed getter/setter logic for catalog_path - Hardcoded catalog path to GloVars.datadir/mcp_catalog.db in: - ProxySQL_MCP_Server.cpp (Query_Tool_Handler initialization) - Admin_FlushVariables.cpp (MySQL_Tool_Handler reinitialization) - Updated VARIABLES.md to document the hardcoded path - Updated configure_mcp.sh to remove catalog_path configuration - Updated MCP README to remove catalog_path references	1 month ago
Rene Cannao	1b7335acfe	Fix two-phase discovery documentation and scripts ... - Add mcp_config.example.json for Claude Code MCP configuration - Fix MCP bridge path in example config (../../proxysql_mcp_stdio_bridge.py) - Update Two_Phase_Discovery_Implementation.md with correct Phase 1/Phase 2 usage - Fix Two_Phase_Discovery_Implementation.md DELETE FROM fts_objects to scope to run_id - Update README.md with two-phase discovery section and multi-agent legacy note - Create static_harvest.sh bash wrapper for Phase 1 - Create two_phase_discovery.py orchestration script with prompts - Add --run-id parameter to skip auto-fetch - Fix RUN_ID placeholder mismatch (<USE_THE_PROVIDED_RUN_ID>) - Fix catalog path default to mcp_catalog.db - Add test_catalog.sh to verify catalog tools work - Fix Discovery_Schema.cpp FTS5 syntax (missing space) - Remove invalid CREATE INDEX on FTS virtual tables - Add MCP tool call logging to track tool usage - Fix Static_Harvester::get_harvest_stats() to accept run_id parameter - Fix DELETE FROM fts_objects to only delete for specific run_id - Update system prompts to say DO NOT call discovery.run_static - Update user prompts to say Phase 1 is already complete - Add --mcp-only flag to restrict Claude Code to MCP tools only - Make FTS table failures non-fatal (check if table exists first) - Add comprehensive documentation for both discovery approaches	1 month ago
Rene Cannao	f9270e6c8b	fix: Correct two_phase_discovery.py usage example in docs ... Add missing --mcp-config argument which is required by the script. The dry-run example now correctly shows all required parameters.	1 month ago
Rene Cannao	6f23d5bcd0	feat: Implement two-phase schema discovery architecture ... Phase 1 (Static/Deterministic): - Add Discovery_Schema: SQLite catalog with deterministic and LLM tables - Add Static_Harvester: MySQL INFORMATION_SCHEMA metadata extraction - Harvest schemas, objects, columns, indexes, foreign keys, view definitions - Compute derived hints: is_time, is_id_like, has_pk, has_fks, has_time - Build quick profiles and FTS5 indexes Phase 2 (LLM Agent): - Add 19 new MCP tools for two-phase discovery - discovery.run_static: Trigger ProxySQL's static harvest - Catalog tools: init, search, get_object, list_objects, get_relationships - Agent tools: run_start, run_finish, event_append - LLM tools: summary_upsert, relationship_upsert, domain_upsert, etc. Files: - include/Discovery_Schema.h, lib/Discovery_Schema.cpp - include/Static_Harvester.h, lib/Static_Harvester.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp (updated) - lib/Makefile (updated) - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/ - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py	1 month ago
Rene Cannao	1193a55e7b	docs: Remove Version History section from LLM Bridge README	1 month ago
Rene Cannao	5afb71ca90	docs: Rename NL2SQL documentation to LLM Bridge ... - Rename doc/NL2SQL/ to doc/LLM_Bridge/ - Update README.md for generic LLM bridge functionality - Replace NL2SQL terminology with LLM Bridge - Update variable references from ai_nl2sql_* to genai_llm_* - Add migration guide from NL2SQL to LLM Bridge - Document new use cases (summarization, code generation, etc.)	1 month ago
Rene Cannao	349320a67f	docs: Fix NL2SQL documentation with genai variables and async architecture ... Updated documentation to reflect the architectural changes: - Changed all ai_* variables to genai-* variables - Changed LOAD MYSQL VARIABLES to LOAD GENAI VARIABLES - Clarified that NL2SQL queries run on MySQL module, not Admin - Updated ARCHITECTURE.md to document async/non-blocking architecture - Added LOAD GENAI VARIABLES TO RUNTIME after variable changes - Changed stats_ai_nl2sql_cache to SHOW STATUS LIKE 'genai-nl2sql%' Key corrections in README.md: - Variables now use genai-* prefix (genai-nl2sql_enabled, etc.) - Commands use LOAD GENAI VARIABLES instead of LOAD MYSQL VARIABLES - Added note that NL2SQL queries are on MySQL module, not Admin interface Key corrections in ARCHITECTURE.md: - Updated system flow diagram to show async socketpair path - Added detailed async flow explanation - Documented that MySQL threads are NOT blocked during LLM calls Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	1 month ago
Rene Cannao	527bfed297	fix: Migrate AI variables to GenAI module for proper architecture ... This commit fixes a serious design flaw where AI configuration variables were not integrated with the ProxySQL admin interface. All ai_* variables have been migrated to the GenAI module as genai-* variables. Changes: - Added 21 new genai_* variables to GenAI_Thread.h structure - Implemented get/set functions for all new variables in GenAI_Thread.cpp - Removed internal variables struct from AI_Features_Manager - AI_Features_Manager now reads from GloGATH instead of internal state - Updated documentation to reference genai-* variables - Fixed debug.cpp assertion for PROXY_DEBUG_NL2SQL and PROXY_DEBUG_ANOMALY Variable mapping: - ai_nl2sql_enabled → genai-nl2sql_enabled - ai_anomaly_detection_enabled → genai-anomaly_enabled - ai_features_enabled → genai-enabled - All other ai_* variables follow the same pattern The flush functions automatically handle all variables in the genai_thread_variables_names array, so database persistence works correctly without additional changes. Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	1 month ago
Rene Cannao	2888ee3f45	Fix gemini-code-assist recommendations and implement comprehensive anomaly detection tests ... - Fix retry logic to use is_retryable_error function for proper HTTP error handling - Add exception handling to get_json_int function with try-catch around std::stoi - Improve validate_numeric_range to use strtol instead of atoi for better error reporting - Fix Chinese characters in documentation (non-zero -> non-zero) - Replace placeholder tests with actual comprehensive tests for anomaly detection functionality - Create new standalone unit test anomaly_detector_unit-t.cpp with 29 tests covering: * SQL injection pattern detection (12 tests) * Query normalization (8 tests) * Risk scoring calculations (5 tests) * Configuration validation (4 tests) - All tests pass successfully, providing meaningful validation of core anomaly detection logic Thanks to gemini-code-assist for the thorough code review and recommendations.	1 month ago
Rene Cannao	8a6b7480b6	docs: Update NL2SQL documentation for v0.2.0 features ... This commit updates the NL2SQL documentation to reflect the new features added in v0.2.0: README.md changes: - Added Request Configuration section with retry parameters - Added Error Handling section with error code table - Added Request Correlation section with log format examples - Updated Results section with error columns - Updated Troubleshooting with retry behavior documentation - Added v0.2.0 to Version History API.md changes: - Updated NL2SQLRequest struct with request_id and retry config fields - Updated NL2SQLResult struct with error details fields - Added NL2SQLErrorCode enum documentation - Updated Result Format with new columns - Expanded Error Codes section with structured error codes TESTING.md changes: - Added Validation Tests to test suite overview - Documented ai_validation-t.cpp test categories - Added instructions for running validation tests - Documented all 61 test cases across 5 categories	1 month ago
Rene Cannao	897d306d2d	Refactor: Simplify NL2SQL to use only generic providers ... Remove Ollama-specific provider code and use only generic OpenAI-compatible and Anthropic-compatible providers. Ollama is now used via its OpenAI-compatible endpoint at /v1/chat/completions. Changes: - Remove LOCAL_OLLAMA from ModelProvider enum - Remove ai_nl2sql_ollama_model and ai_nl2sql_ollama_url variables - Remove call_ollama() function from LLM_Clients.cpp - Update default configuration to use OpenAI provider with Ollama URL - Update all documentation to reflect generic-only approach Configuration: - ai_nl2sql_provider: 'openai' or 'anthropic' (default: 'openai') - ai_nl2sql_provider_url: endpoint URL (default: Ollama OpenAI-compatible) - ai_nl2sql_provider_model: model name - ai_nl2sql_provider_key: API key (optional for local endpoints) This simplifies the codebase by removing a separate code path for Ollama and aligns with the goal of avoiding provider-specific variables.	1 month ago
Rene Cannao	c5a7fc31f7	Add external LLM setup guide and live testing script ... - test_external_live.sh: Interactive script for testing with custom LLM - EXTERNAL_LLM_SETUP.md: Complete guide for external model configuration Covers: - Custom LLM endpoint configuration for NL2SQL - llama-server configuration for embeddings - Architecture overview - Configuration variables - Testing procedures - Troubleshooting tips	1 month ago
Rene Cannao	637b2a669c	feat: Implement NL2SQL vector cache and complete Anomaly threat pattern management ... NL2SQL_Converter improvements: - Implement get_query_embedding() using GenAI module - Implement check_vector_cache() with KNN search via sqlite-vec - Implement store_in_vector_cache() with embedding storage - All stub methods now fully functional Anomaly_Detector improvements: - Implement add_threat_pattern() with embedding generation - Stores patterns in both main table and virtual vec table - Returns pattern ID on success, -1 on error Documentation: - Add comprehensive VECTOR_FEATURES documentation - README.md (471 lines): User guide and quick start - API.md (736 lines): Complete API reference - ARCHITECTURE.md (358 lines): System architecture - TESTING.md (767 lines): Testing guide and procedures This completes the vector features implementation, enabling: - Semantic similarity caching for NL2SQL queries - Embedding-based threat pattern detection - Full CRUD operations for threat patterns	1 month ago
Rene Cannao	0be9715188	test: Add comprehensive tests and documentation for Anomaly Detection ... Added 95 tests (50 unit + 45 integration) and 4 documentation files: Test Files: - test/tap/tests/anomaly_detection-t.cpp (50 unit tests) * Initialization and configuration tests * SQL injection pattern detection * Query normalization * Rate limiting * Statistical anomaly detection * Integration scenarios * Configuration management * False positive handling - test/tap/tests/anomaly_detection_integration-t.cpp (45 integration tests) * Real SQL injection pattern detection with actual queries * Legitimate query passthrough verification * Multi-user rate limiting scenarios * Statistical anomaly detection * Log-only mode configuration Documentation (doc/ANOMALY_DETECTION/): - README.md: User guide with quick start, configuration, examples - API.md: Complete API reference for Anomaly_Detector class - ARCHITECTURE.md: System architecture and design documentation - TESTING.md: Testing guide with test categories and examples All tests compile successfully and follow the TAP framework pattern used throughout ProxySQL.	1 month ago
Rene Cannao	e2d71ec4a2	docs: Add comprehensive NL2SQL user and developer documentation ... User Documentation: - README.md: Complete user guide with examples, configuration, troubleshooting Developer Documentation: - ARCHITECTURE.md: System architecture, components, flow diagrams - API.md: Complete API reference for all variables, structures, and methods - TESTING.md: Testing guide with templates and best practices All documentation follows "very very very" thorough standards with comprehensive examples, diagrams, and cross-references.	1 month ago
Rene Cannao	14de472a3b	Add multi-agent database discovery system ... Implements a 4-agent collaborative system using Claude Code's Task tool and MCP catalog for comprehensive database analysis: - Structural Agent: Maps tables, relationships, indexes, constraints - Statistical Agent: Profiles data distributions, patterns, anomalies - Semantic Agent: Infers business domain and entity types - Query Agent: Analyzes access patterns and optimization Agents collaborate via MCP catalog across 4 rounds: 1. Blind exploration → 2. Pattern recognition → 3. Hypothesis testing → 4. Final synthesis Includes simple_discovery.py demo and comprehensive documentation.	1 month ago
René Cannaò	313f637cf0	Merge branch 'v3.1-vec' into v3.1-MCP1 ... Signed-off-by: René Cannaò <rene.cannao@gmail.com>	1 month ago
Rene Cannao	2ef44e7c3e	Add MCP implementation plans for FTS and Vector Embeddings ... Comprehensive implementation documentation for two new search capabilities: FTS (Full Text Search): - 6 tools for lexical search using SQLite FTS5 - Separate mcp_fts.db database - Keyword matching and phrase search - Tools: fts_index_table, fts_search, fts_list_indexes, fts_delete_index, fts_reindex, fts_rebuild_all Vector Embeddings: - 6 tools for semantic search using sqlite-vec - Separate mcp_embeddings.db database - Vector similarity search with sqlite-rembed integration - Placeholder for future GenAI module - Tools: embed_index_table, embed_search, embed_list_indexes, embed_delete_index, embed_reindex, embed_rebuild_all Both systems: - Follow MySQL_Catalog patterns for SQLite management - Integrate with existing MCP Query endpoint - Work alongside Catalog for AI agent memory - 13-step implementation plans with detailed code examples	1 month ago
Rene Cannao	07dc887af2	Add MCP Tool Discovery Guide ... Comprehensive guide for discovering and using MCP Query endpoint tools: - Tool discovery via tools/list method - Complete list of all Query endpoint tools with parameters - cURL and Python examples for tool discovery and execution - Complete database exploration example - Test script usage guide	1 month ago
Rene Cannao	5846cd8b40	Add Database Discovery Agent architecture documentation ... Comprehensive architecture for AI-powered database discovery agent: - Mixture-of-experts approach (Structural, Statistical, Semantic, Query) - Orchestrator pattern for coordinated exploration - Four-phase discovery process (Blind → Patterns → Hypotheses → Synthesis) - Domain-agnostic design for any database complexity or type - Catalog as shared memory for collaborative expert findings - Multiple real-world examples (law firm, scientific research, ecommerce)	1 month ago
Rene Cannao	ced10dd054	Implement per-endpoint authentication for MCP endpoints ... This commit implements Phase 2 of the MCP multi-endpoint architecture: per-endpoint Bearer token authentication. ## Changes ### lib/MCP_Endpoint.cpp - Implemented `authenticate_request()` method with: - Per-endpoint token validation (mcp-{endpoint}_endpoint_auth) - Bearer token support via Authorization header - Query parameter fallback (?token=xxx) for simple testing - No authentication when token is not configured (backward compatible) - Proper 401 Unauthorized response on auth failure - Token whitespace trimming - Debug logging for troubleshooting ### doc/MCP/Architecture.md - Updated Per-Endpoint Authentication section with complete implementation - Marked Phase 3 authentication task as completed (✅) - Added authentication implementation code example ## Authentication Flow 1. Client sends request with Bearer token: - Header: `Authorization: Bearer <token>` - Or query param: `?token=<token>` 2. Server validates against endpoint-specific variable: - `/mcp/config` → `mcp-config_endpoint_auth` - `/mcp/observe` → `mcp-observe_endpoint_auth` - `/mcp/query` → `mcp-query_endpoint_auth` - `/mcp/admin` → `mcp-admin_endpoint_auth` - `/mcp/cache` → `mcp-cache_endpoint_auth` 3. Returns 401 Unauthorized if: - Auth is required but not provided - Token doesn't match expected value 4. Allows request if: - No auth token configured (backward compatible) - Token matches expected value ## Testing ```bash # Set auth token for /mcp/query endpoint mysql -h 127.0.0.1 -P 6032 -u admin -padmin \ -e "SET mcp-query_endpoint_auth='my-secret-token'; LOAD MCP VARIABLES TO RUNTIME;" # Test with Bearer token curl -k -X POST https://127.0.0.1:6071/mcp/query \ -H "Content-Type: application/json" \ -H "Authorization: Bearer my-secret-token" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' # Test with query parameter curl -k -X POST "https://127.0.0.1:6071/mcp/query?token=my-secret-token" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' ``` ## Status ✅ Authentication fully implemented and functional ⚠️ Testing with running ProxySQL instance still needed Co-authored-by: Claude <claude@anthropic.com>	1 month ago
Rene Cannao	c86a048d9c	Implement MCP multi-endpoint architecture with dedicated tool handlers ... This commit implements Option 1 (Multiple Tool Handlers) for the MCP module, where each of the 5 endpoints has its own dedicated tool handler with specific tools. ## Architecture Changes - Created MCP_Tool_Handler base class interface for all tool handlers - Each endpoint now has its own dedicated tool handler: - /mcp/config → Config_Tool_Handler (configuration management) - /mcp/query → Query_Tool_Handler (database exploration) - /mcp/admin → Admin_Tool_Handler (administrative operations) - /mcp/cache → Cache_Tool_Handler (cache management) - /mcp/observe → Observe_Tool_Handler (monitoring & metrics) ## New Files Base Interface: - include/MCP_Tool_Handler.h - Base class for all tool handlers Tool Handlers: - include/Config_Tool_Handler.h, lib/Config_Tool_Handler.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp - include/Admin_Tool_Handler.h, lib/Admin_Tool_Handler.cpp - include/Cache_Tool_Handler.h, lib/Cache_Tool_Handler.cpp - include/Observe_Tool_Handler.h, lib/Observe_Tool_Handler.cpp Documentation: - doc/MCP/Architecture.md - Comprehensive architecture documentation ## Modified Files - include/MCP_Thread.h, lib/MCP_Thread.cpp - Added 5 tool handler pointers - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - Use tool_handler base class - lib/ProxySQL_MCP_Server.cpp - Create and pass handlers to endpoints - lib/Makefile - Added new source files ## Implementation Status - Config_Tool_Handler: Functional (get_config, set_config, list_variables, get_status) - Query_Tool_Handler: Functional (wraps MySQL_Tool_Handler, all 18 tools) - Admin_Tool_Handler: Stub implementations (TODO: implement) - Cache_Tool_Handler: Stub implementations (TODO: implement) - Observe_Tool_Handler: Stub implementations (TODO: implement) See GitHub Issue #8 for detailed TODO list. Co-authored-by: Claude <claude@anthropic.com>	1 month ago
Rene Cannao	a5f712e7d9	Add MCP variables documentation ... Added comprehensive documentation for all 14 MCP module configuration variables: Server Configuration: - mcp-enabled (boolean, default: false) - mcp-port (integer, default: 6071) - mcp-timeout_ms (integer, default: 30000) Endpoint Authentication (5 variables): - mcp-config_endpoint_auth (string, default: "") - mcp-observe_endpoint_auth (string, default: "") - mcp-query_endpoint_auth (string, default: "") - mcp-admin_endpoint_auth (string, default: "") - mcp-cache_endpoint_auth (string, default: "") MySQL Tool Handler Configuration (5 variables): - mcp-mysql_hosts (string, default: "127.0.0.1") - mcp-mysql_ports (string, default: "3306") - mcp-mysql_user (string, default: "") - mcp-mysql_password (string, default: "") - mcp-mysql_schema (string, default: "") Catalog Configuration: - mcp-catalog_path (string, default: "mcp_catalog.db") Includes documentation for management commands, variable persistence, status variables, and security considerations.	1 month ago
Rene Cannao	b77d38c2ca	Add comprehensive GenAI module documentation ... Add doc/GENAI.md documenting the GenAI (Generative AI) module's current state, architecture, configuration, and usage. Documentation includes: - Module overview and version info - Async architecture diagrams (socketpair + epoll) - Communication protocol structures (request/response headers) - Configuration variables (threads, URIs, timeouts) - GENAI: query syntax and examples - Integration with ProxySQL main loop - Backend service integration (llama-server) - Testing guide with TAP test suite - Performance characteristics and resource usage - Error handling and troubleshooting - Platform requirements and limitations - Future enhancements roadmap - Related documentation links	1 month ago
Rene Cannao	8c90bda52a	Address gemini-code-assist review comments for SSL keylog documentation ... This commit addresses all review comments from gemini-code-assist on PR #5279: 1. Fixed FLUSH LOGS documentation - clarified that file is reopened for appending, not truncating, and updated the note about preserving contents 2. Fixed callback documentation - clarified that the callback attaches to all frontend connections, not just admin connections 3. Updated security warning - focused on passive eavesdropping and offline decryption as the primary threats 4. Fixed typo: proxyql_ip -> proxysql_ip in tcpdump example 5. Removed misleading @see HPKP link - HPKP is unrelated to NSS Key Log Format and is a deprecated feature 6. Updated NSS Key Log Format URL to use official MDN link instead of unofficial mirror 7. Fixed buffer size comment to accurately reflect 256-byte buffer and 254-byte line length validation 8. Clarified fputs comment to emphasize the read lock's role in allowing concurrent writes from multiple threads	2 months ago
Rene Cannao	442635b721	Add comprehensive documentation for SSL/TLS key logging feature ... This commit adds extensive documentation for the ssl_keylog_file feature (introduced in PR #4236), which enables TLS key logging for debugging encrypted traffic. ## Background The ssl_keylog_file variable (exposed as admin-ssl_keylog_file in SQL interface) allows ProxySQL to write TLS secrets to a file in NSS Key Log Format. These secrets can be used by tools like Wireshark and tshark to decrypt and analyze TLS traffic for debugging purposes. ## Changes ### Inline Documentation (Code) 1. include/proxysql_sslkeylog.h (+96 lines) - File-level documentation explaining the module purpose and security - Doxygen comments for all 5 public APIs - Thread-safety annotations - Parameter descriptions and return values 2. lib/proxysql_sslkeylog.cpp (+136 lines) - Implementation-level documentation - Algorithm explanations (double-checked locking, thread safety) - Reference to NSS Key Log Format specification 3. include/proxysql_admin.h (+19 lines) - Variable documentation for ssl_keylog_file - Path handling rules (absolute vs relative) - Security implications ### Developer Documentation (doc/ssl_keylog/ssl_keylog_developer_guide.md) Target audience: Developers working on ProxySQL codebase Contents: - Variable naming convention (SQL vs config file vs internal) - Architecture diagrams - Thread safety model (pthread rwlock) - NSS Key Log Format specification - Complete API reference for all public functions - Integration points in the codebase - Security considerations and code review checklist - Testing procedures ### User Documentation (doc/ssl_keylog/ssl_keylog_user_guide.md) Target audience: End users and system administrators Contents: - What is SSL key logging and when to use it - Variable naming: admin-ssl_keylog_file (SQL) vs ssl_keylog_file (config) - Step-by-step enable/disable instructions - Path resolution (absolute vs relative) - Log rotation procedures - Production workflow: tcpdump capture → offline analysis - Wireshark (GUI) integration tutorial - tshark (command-line) usage examples - Troubleshooting common issues - Security best practices - Quick reference card ## Key Features Documented 1. **Variable Naming Convention** - SQL interface: SET admin-ssl_keylog_file = '/path'; - Config file: ssl_keylog_file='/path' (in admin_variables section) - Internal code: ssl_keylog_file 2. **Production Workflow** - Capture traffic with tcpdump (no GUI on production server) - Transfer pcap + keylog to analysis system - Analyze offline with Wireshark (GUI) or tshark (CLI) 3. **tshark Examples** - Command-line analysis of encrypted traffic - Filter examples for debugging TLS issues - JSON export for automated analysis ## Security Notes The documentation emphasizes that: - Key log files contain cryptographic secrets that decrypt ALL TLS traffic - Access must be restricted (permissions 0600) - Only enable for debugging, never in production - Securely delete old key log files ## Files Modified - include/proxysql_admin.h - include/proxysql_sslkeylog.h - lib/proxysql_sslkeylog.cpp ## Files Added - doc/ssl_keylog/ssl_keylog_developer_guide.md - doc/ssl_keylog/ssl_keylog_user_guide.md	2 months ago
Wazir Ahmed	facba20055	doc: Add information about groups in TAP test guide ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Rene Cannao	cbf27eb607	Add vec0 KNN LIMIT constraint documentation for Posts embeddings ... Add documentation for common error "A LIMIT or 'k = ?' constraint is required on vec0 knn queries" with corrected query examples. Changes: 1. Add Example 3 showing correct LIMIT placement for "What is ProxySQL?" query 2. Include alternative syntax using 'k = ?' constraint 3. Explain key rules for vec0 KNN queries: - LIMIT or k = ? must be in same query level as MATCH - Cannot use both constraints together - When joining, put MATCH + LIMIT in subquery - Constraint tells sqlite-vec how many similar vectors to return	2 months ago
Wazir Ahmed	67ca39e248	doc: Update TAP documentation with unit test instructions ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Rene Cannao	8e83635760	Add Posts embeddings setup documentation with optimized batch processing ... This documentation provides step-by-step instructions for setting up virtual tables for Posts embeddings using sqlite-rembed and sqlite-vec. Key features: - Virtual table creation for 768-dimensional embeddings - API client configuration for embedding generation - Optimized batch query using LEFT JOIN to process unembedded rows - Batch processing script with progress tracking - Similarity search examples - Performance considerations and troubleshooting The batch query uses a LEFT JOIN pattern to find unprocessed rows: INSERT ... SELECT ... FROM Posts LEFT JOIN Posts_embeddings WHERE Posts_embeddings.rowid IS NULL LIMIT 1000; This approach eliminates the need for tracking rowid ranges and can be run repeatedly until all 248,905 Posts have embeddings generated.	2 months ago