1 Commits (feature/pgsql-native-backend-protocol)

Author	SHA1	Message	Date
Rene Cannao	bcefd093fc	test(genai): add live TAP validations for GenAI, LLM bridge, and MCP semantic lifecycle ... This commit introduces three new C++ TAP tests that validate ProxySQL's live GenAI and MCP behaviors using real provider credentials supplied via environment variables. The goal is to move beyond mock-style checks and verify actual runtime integration across request transport, tool execution, and semantic outputs. High-level scope - Add a live GenAI embed/rerank validation TAP test. - Add a live LLM bridge accuracy/error-path TAP test. - Add a live MCP semantic lifecycle TAP test that combines discovery, LLM-generated artifacts, upsert, and semantic search. - Register all three new tests in ai-g1 group mapping. Files added - test/tap/tests/genai_live_validation-t.cpp - test/tap/tests/llm_bridge_accuracy-t.cpp - test/tap/tests/mcp_semantic_lifecycle-t.cpp File updated - test/tap/groups/groups.json Detailed behavior by test 1) genai_live_validation-t.cpp - Reads required live env inputs: TAP_EMBED_URL, TAP_EMBED_TYPE, TAP_EMBED_MODEL, TAP_EMBED_DIMENSION, TAP_RERANK_URL, TAP_RERANK_MODEL - Skips (does not fail) when required environment is missing. - Configures runtime for test stability: - sets genai-vector_db_path to ./ai_features.db - enables genai-enabled - sets embed/rerank endpoints and embedding model - loads GENAI variables to runtime - Embedding integrity validation: - sends GENAI embed request with multiple documents - verifies row count matches input document count - verifies each returned embedding dimension matches TAP_EMBED_DIMENSION - Rerank semantic validation: - sends query with one intentionally relevant document and irrelevant distractors - checks highest score maps to the relevant document index - Stress validation: - opens 5 client connections - executes 20 total requests (4 per connection) concurrently - validates all requests succeed and no failures are reported - Includes high-verbosity diagnostics for SQL requests and parsed rows. 2) llm_bridge_accuracy-t.cpp - Reads required live env inputs: TAP_LLM_PROVIDER, TAP_LLM_URL, TAP_LLM_MODEL, TAP_LLM_KEY - Skips (does not fail) when required environment is missing. - Configures LLM bridge runtime: - sets genai-vector_db_path - enables genai-enabled and genai-llm_enabled - sets provider/url/model/key - loads GENAI variables to runtime - Special-character prompt handling: - issues LLM: prompt containing quotes, backslashes, JSON-like text, and emoji bytes - verifies request succeeds and response structure is valid - verifies returned provider column aligns with TAP_LLM_PROVIDER - Timeout/error-path validation: - reconfigures provider URL to an unroutable timeout-probe endpoint - sets genai-llm_timeout_ms=1000 (minimum valid bound in current code) - verifies client receives an error path with SQLSTATE HY000 and non-empty message - Captures and restores modified global variables at end of test. 3) mcp_semantic_lifecycle-t.cpp - Reads required live env inputs: TAP_LLM_PROVIDER, TAP_LLM_URL, TAP_LLM_MODEL, TAP_LLM_KEY - Skips (does not fail) when required environment is missing. - Configures LLM and MCP runtime for end-to-end lifecycle checks: - enables genai + llm bridge - configures MCP port/auth endpoint settings - creates MCP auth and target profiles - loads MCP variables/profiles to runtime - End-to-end lifecycle: - calls discovery.run_static and validates run_id - lists discovered table objects and selects object_ids - starts agent run via agent.run_start - generates summary text through LLM: bridge for two semantic markers - persists summaries via llm.summary_upsert for two objects - validates llm.search("customer") finds customer marker - validates llm.search("index") finds index marker - finishes run via agent.run_finish - Cleans up test MCP profiles and restores runtime variables. Groups registration - Added to ai-g1 in test/tap/groups/groups.json: - llm_bridge_accuracy-t - genai_live_validation-t - mcp_semantic_lifecycle-t Implementation notes and constraints reflected in tests - Tests are intentionally environment-gated and skip when live credentials are unavailable. - All tests include verbose diagnostics for outbound requests and parsed provider/tool responses. - Runtime variable mutations are restored best-effort to reduce suite side effects. - llm timeout validation uses 1000ms because current runtime validation enforces [1000..600000] for genai-llm_timeout_ms. Build verification performed - Compiled successfully (as jenkins user): - genai_live_validation-t - llm_bridge_accuracy-t - mcp_semantic_lifecycle-t This commit intentionally focuses on live integration correctness and transport behavior under real endpoints, while remaining TAP-friendly for CI environments that may not provide credentials (skip semantics instead of hard failures).	4 months ago