proxysql

Commit Graph

Author	SHA1	Message	Date
Wazir Ahmed	9757835a5f	Merge branch 'v3.0' into fix-fc-parsing	2 months ago
Rene Cannao	76251a78dc	fix: resolve invalidFunctionArg in get_current_query - use size_t cast	2 months ago
Rene Cannao	d5ffcba695	fix: cppcheck deeper analysis - invalidLifetime and arrayIndexOutOfBounds - Fix invalidLifetime: look up pointer from map after insertion, not before - Fix arrayIndexOutOfBounds: add bounds check before array access	2 months ago
Rene Cannao	bf9030ecb7	fix: remaining cppcheck printf and null pointer issues - Fix wrongPrintfScanfArgNum: remove extra args in 4 sprintf calls - Fix nullPointer: add GloMTH check, add null checks for strdup args - All high-risk cppcheck warnings now resolved	2 months ago
Rene Cannao	5ccb715e57	fix: cppcheck fixes - printf formats, null checks, copy operators - Fix invalidPrintfArgType: %d->%u for hid, %u->%d for use_ssl - Fix nullPointerRedundantCheck: replace assert with continue - Add deleted copy ctor/operator= to KillArgs and MySQL_Session - Fix invalidFunctionArg: add query_len > 3 check before memcpy	2 months ago
Rene Cannao	b5617c0c18	fix: initialize all members in Query_Info constructor	2 months ago
Wazir Ahmed	0e5fd66d46	Preserve prepared statement `min_gtid` in `first_comment_parsing` mode Issue --- When `first_comment` parsing runs before query rule processing, query rules can strip the `min_gtid` annotation during `STMT_PREPARE`. As a result, `STMT_EXECUTE` can run without the original `min_gtid` constraint. Refs --- - Commit - `1c25787b88` - https://github.com/sysown/proxysql/issues/5384#issuecomment-4137207612 Fix --- - Store `min_gtid` by frontend `client_stmt_id` in `MySQL_STMTs_local_v14` and restore it during `STMT_EXECUTE` using the client statement handle. - Add TAP test to cover the case of `[first_comment_parsing=1 + prepared_stmt + min_gtid + query_rewrite]` Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
René Cannaò	3659ba85fb	Merge branch 'v3.0' into advisory/com-stmt-send-long-data-fix-1 Signed-off-by: René Cannaò <rene@proxysql.com>	2 months ago
Rene Cannao	151169da9b	Fix COM_STMT_SEND_LONG_DATA short packet handling	2 months ago
René Cannaò	419cd7608b	Merge commit from fork Fix COM_STMT_CLOSE short packet handling	2 months ago
Rene Cannao	b2f4b4f9a0	Fix crash when COM_BINLOG_DUMP received with idle backends from other hostgroups (#5556 ) When a session has backend connections to multiple hostgroups and receives COM_BINLOG_DUMP, GPFC_Replication_SwitchToFastForward() transitions the session to FAST_FORWARD mode but only manages the active backend. Idle backends from other hostgroups remain in the session's mybes array with async_state_machine==ASYNC_IDLE. If MySQL's wait_timeout closes one of these idle connections, the resulting POLLIN event causes handler() to be called with ASYNC_IDLE, which has no case in the switch and hits assert(0). Fix: Clean up idle backends from other hostgroups during the FAST_FORWARD transition in GPFC_Replication_SwitchToFastForward(). These connections will never be used again, so they are destroyed immediately. Includes a regression test (test_binlog_dump_multi_backend_crash-t) that: - Sets up two hostgroups with query rules routing SELECTs to reader HG - Creates backend connections to both hostgroups on one session - Sends COM_BINLOG_DUMP to enter FAST_FORWARD mode - Waits for wait_timeout to close the idle backend - Verifies ProxySQL does not crash	2 months ago
Rene Cannao	9450bf52e2	Fix COM_STMT_CLOSE short packet handling	2 months ago
Rene Cannao	6b46b26931	chore: implement PR review feedback for release tier system - Makefile: normalized GIT_VERSION by stripping 'v' prefix before arithmetic. - Makefile: corrected .PHONY declaration for build_deps_debug_default. - Admin Handler: properly guarded TSDB commands to return error when disabled. - REST API: removed redundant nested guards inside tsdb_resource class. - Statistics: implemented rigorous input validation for set_variable. - Statistics: added return value checks and rollback logic for TSDB migrations. - Statistics: implemented bounded concurrency (batching) for backend probes. - FFTO: implemented cumulative buffer limit enforcement in MySQL and PgSQL. - FFTO: ensured initial packet capture in PgSQL_Session by early initialization. - FFTO: improved reassembly buffer compaction logic. - Headers: default-initialized ffto_bypassed to false. - Code cleanup: merged redundant PROXYSQLFFTO guards.	3 months ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	3 months ago
Rene Cannao	e53d2c76f9	Address PR #5410 review findings across FFTO, sessions, TAP, and docs Apply actionable bot review feedback from PR #5410 without behavior drift:\n- fix MySQL/PgSQL FFTO digest normalization and max-length hashing\n- add early FFTO packet/message size guards for bypass safety\n- restore MySQL session GTID reset and harden GenAI async error ownership\n- add null-check for stripped KILL query normalization path\n- call FFTO on_close() before observer reset/bypass in session flows\n- fix processlist query-options default pagination limit semantics\n- fix TAP noise_failures locking race and noise-aware plan count\n- extend wait_get_enpoint_ready with optional basic-auth passthrough\n- update MCP architecture docs from observe->stats naming\n- clean corrupted character in pgsql query-cache test comment\n\nValidation run:\n- git diff --check\n- make -C test/tap/tap tap.o utils_mariadb.o utils_mysql57.o utils_mysql8.o -j4\n- make -C lib obj/MySQLFFTO.oo obj/PgSQLFFTO.oo obj/MySQL_Session.oo obj/PgSQL_Session.oo obj/Query_Processor.oo -j4	3 months ago
Rene Cannao	79c0b383e8	integration: fix pre-CI issues in unified branch	3 months ago
Rene Cannao	90462f6d5a	ffto: harden MySQL/PgSQL observer state handling and align tests/docs Address outstanding review findings for FFTO on v3.0-ff_inspect and tighten protocol-state correctness for both engines. MySQL FFTO - Restrict on_close() reporting to true in-flight states and always clear query tracking state after close. - Add explicit active-query cleanup helpers and invoke them on state transitions to IDLE. - Preserve accounting on mid-resultset server ERR packets by reporting current query in READING_COLUMNS/READING_ROWS before reset. - Keep prepared-statement lifecycle cleanup robust (pending prepare cleared on prepare completion paths). MySQL session integration - Extract duplicated FAST_FORWARD client FFTO feed logic into observe_ffto_client_packet() and reuse it from all call sites. PostgreSQL FFTO - Replace regex-based CommandComplete parsing with lightweight token parsing, including NUL/whitespace trimming and strict numeric validation. - Add queued tracking for pipelined extended-protocol executes so query text and response attribution stay aligned under Parse/Bind/Execute pipelining. - Distinguish finalize semantics (execute-finalize on CommandComplete vs sync-finalize on ReadyForQuery) and centralize finalize/activation helpers. - Add frontend Close ('C') handling to evict statement/portal mappings. - Harden client/server message parsing with additional length checks. - Extend affected-row command tag coverage to COPY and MERGE. TAP tests - Stabilize test plans for failure paths by replacing early returns with a fail-and-skip-remaining flow and shared cleanup labels. - Ensure both MySQL and PgSQL FFTO tests preserve planned assertion counts under setup/prepare/execute failures. Documentation - Align FFT0 design doc state/response descriptions with current implementation (ReadyForQuery handling, pipelined queueing, supported PG command tags). - Fix wording/typo issues in protocol section. Validation performed - make -C lib -j4 - make -C test/tap/tests test_ffto_mysql-t test_ffto_pgsql-t -j4 Runtime execution of the two TAP binaries remains environment-dependent (admin endpoint connectivity required).	3 months ago
Rene Cannao	1258a1f2f2	Address PR reviews: improve performance, safety, and robustness of FFTO	3 months ago
Rene Cannao	db04a54c92	Fix memory corruption and stack overflow in FFTO due to large queries	3 months ago
Rene Cannao	3a3be5a559	Implement Fast Forward Traffic Observer (FFTO) for MySQL and PostgreSQL - Added TrafficObserver base interface - Implemented MySQLFFTO and PgSQLFFTO classes for protocol-aware observation - Integrated FFTO hooks into MySQL and PostgreSQL session handlers in FAST_FORWARD state - Added configuration variables: mysql-ffto_enabled, mysql-ffto_max_buffer_size, pgsql-ffto_enabled, pgsql-ffto_max_buffer_size - Implemented query digest extraction and reporting parity with internal stats - Added session-level bypass when packet size exceeds buffer threshold - Included architectural documentation in doc/ffto_design.md	3 months ago
René Cannaò	3b578db4bf	Merge pull request #5374 from sysown/v3.0-test0213 Fix Valgrind-reported uninitialized memory issues	3 months ago
Rene Cannao	af58865b06	Fix uninitialized memory in QueryParserArgs.buf Use memset to zero-initialize the entire QueryParserArgs struct instead of individually setting only digest_text and first_comment to NULL. This fixes Valgrind errors about use of uninitialised values in strcasestr() when reading from the uninitialized buf array. Fixes errors like: Use of uninitialised value of size 8 at tolower (ctype.c:46) by strcasestr by MySQL_Connection::ProcessQueryAndSetStatusFlags(char*)	3 months ago
Rene Cannao	41564acaac	Fix #5355 : Add null pointer check in RequestEnd() to prevent use-after-free crash This commit addresses a use-after-free race condition that could cause ProxySQL to crash with SIGSEGV during COM_CHANGE_USER timeout scenarios. Root Cause: ----------- When a session is in RESETTING_CONNECTION status and the backend times out during COM_CHANGE_USER, the code calls RequestEnd() to handle the error. However, RequestEnd() accesses client_myds without null checks. If the session is marked as unhealthy and deleted by ProcessAllSessions_Healthy0() while RequestEnd() is still executing, this results in a use-after-free crash. The crash is particularly likely during AWS Aurora failures when: 1. Monitor detects Aurora master timeouts 2. Multiple sessions experience connection issues 3. Many client connections are marked unhealthy 4. The high volume of session deletions increases race condition probability Fix: ---- Added a null pointer check before accessing client_myds in the unexp_com_pings handling code within RequestEnd(). This is a defensive programming fix that prevents the crash when client_myds has been freed. A more robust fix would involve proper session lifecycle management with reference counting. Fixes: #5355	4 months ago
René Cannaò	de0a9b9c27	Merge pull request #5093 from Gonlo2/fix-usage-deleted-con Avoid usage of deleted connection	4 months ago
Rene Cannao	48bc7dd7bf	Merge v4.0 GenAI features into v3.0 with conditional compilation This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)	4 months ago
Rene Cannao	c914feb230	Fix security issues identified in PR #5312 code review This commit addresses critical and important security vulnerabilities found during comprehensive code review of the Gen AI features merge. Critical fixes: - SQL injection vulnerabilities in MySQL_Tool_Handler.cpp: - Added validate_sql_identifier() for schema/table validation - Added escape_string() for MySQL string escaping using mysql_real_escape_string - Fixed list_tables(), describe_table(), sample_rows(), sample_distinct() - SQL injection vulnerabilities in Query_Tool_Handler.cpp: - Added validate_sql_identifier_sqlite() for identifier validation - Added escape_string_literal() for SQLite string escaping - Fixed list_tables tool and catalog.get_relationships function - Use-after-free race condition in GenAI_Thread: - Changed shutdown_ from int to std::atomic<int> for proper memory ordering - Added additional shutdown check in worker_loop after popping request Important fixes: - Buffer overflow risks from sprintf usage: - Converted all sprintf() calls to snprintf() in GenAI_Thread.cpp - Converted sprintf() to snprintf() in MySQL_Session.cpp - Worker loop shutdown race condition: - Added shutdown check after popping request from queue - Properly clean up client_fd when shutdown is detected These fixes ensure: 1. All user input is properly validated before use in SQL queries 2. String values are properly escaped using database-specific escaping 3. Thread-safe shutdown with proper memory ordering guarantees 4. Bounds-safe string formatting to prevent buffer overflows	4 months ago
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec Signed-off-by: René Cannaò <rene@proxysql.com>	4 months ago
Rene Cannao	a3f0bade4e	feat: Convert NL2SQL to generic LLM bridge - Rename NL2SQL_Converter to LLM_Bridge for generic prompt processing - Update MySQL protocol handler from /* NL2SQL: / to / LLM: / - Remove SQL-specific fields (sql_query, confidence, tables_used) - Add GENAI_OP_LLM operation type to GenAI module - Rename all genai_nl2sql_ variables to genai_llm_* - Update AI_Features_Manager to use LLM_Bridge - Deprecate ai_nl2sql_convert MCP tool with error message - LLM bridge now handles any prompt type via MySQL protocol This enables generic LLM access (summarization, code generation, translation, analysis) while preserving infrastructure for future NL2SQL implementation via Web UI + external agents.	4 months ago
Rene Cannao	a7dac5ef3d	feat: Make NL2SQL use async GenAI path instead of blocking calls This is a critical architectural fix - NL2SQL was making blocking calls to LLMs which would block the entire MySQL thread. Now NL2SQL uses the same async socketpair pattern as the GENAI embed/rerank operations. Changes: - Added nl2sql operation type to process_json_query() in GenAI module - Updated NL2SQL handler to construct JSON query and use async GENAI path - Added extern declaration for GloAI in GenAI_Thread.cpp - Falls back to synchronous path only on systems without epoll Architecture: - Before: NL2SQL: query → blocking nl2sql->convert() → blocks MySQL thread - After: NL2SQL: query → JSON GENAI request → async socketpair → non-blocking JSON protocol for NL2SQL: GENAI: {"type": "nl2sql", "query": "Show customers", "schema": "mydb"} The NL2SQL result is delivered asynchronously through the existing GENAI response handler, making the system fully non-blocking. Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	4 months ago
Rene Cannao	ae4200dbc0	Enhance AI features with improved validation, memory safety, error handling, and performance monitoring - Rename validate_provider_name to validate_provider_format for clarity - Add null checks and error handling for all strdup() operations - Enhance error messages with more context and HTTP status codes - Implement performance monitoring with timing metrics for LLM calls and cache operations - Add comprehensive test coverage for edge cases, retry scenarios, and performance - Extend status variables to track performance metrics - Update MySQL session to report timing information to AI manager	4 months ago
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. Core Detection Methods: 1. SQL Injection Pattern Detection (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. Query Normalization (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. Rate Limiting (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. Statistical Anomaly Detection (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. Embedding-based Similarity (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations Query Flow Integration: - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details Status Variables Added: - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` Configuration (defaults): - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false Detection Pipeline: ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` Files Modified: - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions Next Steps: - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	4 months ago
Rene Cannao	bc4fff12ce	feat: Add NL2SQL query interception in MySQL_Session - Add NL2SQL handler declaration - Add routing for 'NL2SQL:' prefix - Return resultset with generated SQL and metadata	4 months ago
Rene Cannao	33a87c66a7	Fix critical issues identified by gemini-code-assist This commit addresses critical issues identified in the code review: 1. Fix non-blocking read handling: - lib/GenAI_Thread.cpp (listener_loop): Properly handle EAGAIN/EWOULDBLOCK - Return early on EAGAIN/EWOULDBLOCK instead of closing connection - Handle EOF (n==0) separately from errors (n<0) - lib/MySQL_Session.cpp (handle_genai_response): Properly handle EAGAIN/EWOULDBLOCK - Return early on EAGAIN/EWOULDBLOCK instead of cleaning up request - Use goto for cleaner control flow 2. Refactor JSON building/parsing to use nlohmann/json: - lib/GenAI_Thread.cpp (call_llama_batch_embedding): - Replace manual stringstream JSON building with nlohmann/json - Replace fragile string-based parsing with nlohmann/json::parse() - Support multiple response formats (results, data, embeddings) - Add proper error handling with try/catch - lib/GenAI_Thread.cpp (call_llama_rerank): - Replace manual stringstream JSON building with nlohmann/json - Replace fragile string-based parsing with nlohmann/json::parse() - Support multiple response formats and field names - Add proper error handling with try/catch These changes: - Fix potential connection drops due to incorrect EAGAIN handling - Improve security and robustness of JSON handling - Reduce code complexity and improve maintainability - Add support for multiple API response formats	5 months ago
Rene Cannao	db2485be37	Add comprehensive doxygen documentation to GenAI async module This commit adds extensive doxygen-format documentation to all key functions in the GenAI async module to improve code maintainability and API clarity. Documented functions: - lib/GenAI_Thread.cpp: - unregister_client() - cleanup flow and usage - call_llama_batch_embedding() - HTTP client with JSON format - call_llama_rerank() - HTTP client with JSON format - execute_sql_for_documents() - stub for document_from_sql - process_json_query() - autonomous JSON query processing - lib/MySQL_Session.cpp: - genai_send_async() - async flow and error handling - handle_genai_response() - response handling flow - genai_cleanup_request() - resource cleanup details - check_genai_events() - main loop integration Enhanced header documentation: - GenAI_RequestHeader - communication flow details - GenAI_ResponseHeader - response format details - register_client() - registration flow - unregister_client() - cleanup flow - embed_documents() - BLOCKING warning - rerank_documents() - BLOCKING warning - process_json_query() - supported formats All documentation includes: - @brief descriptions - @param parameter details - @return return value explanations - @note important warnings and usage notes - @see cross-references to related functions - Detailed workflow descriptions - Error handling details - Memory management notes	5 months ago
Rene Cannao	8405027124	Integrate GenAI async event handling into main MySQL session loop - Add check_genai_events() function for non-blocking epoll_wait on GenAI response fds - Integrate GenAI event checking into main handler() WAITING_CLIENT_DATA case - Add goto handler_again to process multiple GenAI responses in one iteration The async GenAI architecture is now fully integrated. MySQL threads no longer block when processing GENAI: queries - they send requests asynchronously via socketpair and continue processing other queries while GenAI workers handle the embedding/reranking operations.	5 months ago
Rene Cannao	0ff2e38e22	Implement async GenAI module with socketpair-based non-blocking architecture - Add GenAI_RequestHeader and GenAI_ResponseHeader protocol structures for socketpair communication - Implement GenAI listener_loop to read requests from epoll and queue to workers - Implement GenAI worker_loop to process requests and send responses via socketpair - Add GenAI_PendingRequest state management to MySQL_Session/Base_Session - Implement MySQL_Session async handlers: genai_send_async(), handle_genai_response(), genai_cleanup_request() - Modify MySQL_Session genai handler to use async path when epoll is available - Initialize GenAI epoll fd in Base_Session::init() This completes the async architecture that was planned but never fully implemented (previously had only placeholder comments). The GenAI module now processes requests asynchronously without blocking MySQL threads.	5 months ago
Rene Cannao	a82f58e22b	Refactor GenAI module for autonomous JSON query processing Move all JSON parsing and operation routing logic from MySQL_Session to GenAI module. MySQL_Session now simply passes GENAI: queries to the GenAI module via process_json_query(), which handles everything autonomously. This simplifies the architecture and achieves better separation of concerns: - MySQL_Session: Detects GENAI: prefix and forwards to GenAI module - GenAI module: Handles JSON parsing, operation routing, and result formatting Changes: - GenAI_Thread.h: Add GENAI_OP_JSON operation type, json_query field, and process_json_query() method declaration - GenAI_Thread.cpp: Implement process_json_query() with embed/rerank support and document_from_sql framework (stubbed for future MySQL connection handling) - MySQL_Session.cpp: Simplify genai handler to just call process_json_query() and parse JSON result (reduces net code by ~215 lines)	5 months ago
Rene Cannao	cc3e97b7b8	Merge EMBED and RERANK into unified GENAI: query syntax This commit refactors the experimental GenAI query syntax to use a single GENAI: keyword with type-based operations instead of separate EMBED: and RERANK: keywords. Changes: - Replace EMBED: and RERANK: detection with unified GENAI: detection - Merge genai_embedding and genai_rerank handlers into single genai handler - Add 'type' field to operation JSON ("embed" or "rerank") - Add 'columns' field for rerank operation (2 or 3, default 3) - columns=2: Returns only index and score - columns=3: Returns index, score, and document (default) Old syntax: EMBED: ["doc1", "doc2"] RERANK: {"query": "...", "documents": [...], "top_n": 5} New syntax: GENAI: {"type": "embed", "documents": ["doc1", "doc2"]} GENAI: {"type": "rerank", "query": "...", "documents": [...], "top_n": 5, "columns": 2} This provides a cleaner, more extensible API for future GenAI operations.	5 months ago
Rene Cannao	39939f598b	Add experimental GenAI RERANK: query support for MySQL This commit adds experimental support for reranking documents directly from MySQL queries using a special RERANK: syntax. Changes: - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_rerank() - Add RERANK: query detection alongside EMBED: detection - Implement JSON parsing for query, documents array, and optional top_n - Build resultset with index, score, and document columns - Use MySQL ERR_Packet for error handling Query format: RERANK: {"query": "search query", "documents": ["doc1", "doc2", ...], "top_n": 5} Result format: 1 row per result, 3 columns (index, score, document)	5 months ago
Rene Cannao	253591d262	Add experimental GenAI EMBED: query support for MySQL This commit adds experimental support for generating embeddings directly from MySQL queries using a special EMBED: syntax. Changes: - Add MYDS_INTERNAL_GENAI to MySQL_DS_type enum for GenAI connections - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_embedding() - Implement EMBED: query detection and JSON parsing for document arrays - Build CSV resultset with embeddings (1 row per document, 1 column) - Add myconn NULL check in MySQL_Thread for INTERNAL_GENAI type - Add "debug_genai" name to debug module array - Remove HAVE_LIBCURL checks (libcurl is always statically linked) - Use static curl header: "curl/curl.h" instead of <curl/curl.h> - Remove curl_global_cleanup() from GenAI module (should only be in main()) Query format: EMBED: ["doc1", "doc2", ...] Result format: 1 row per document, 1 column with CSV embeddings Error handling uses MySQL ERR_Packet instead of resultsets.	5 months ago
Rene Cannao	fc73ec1c50	Code review improvements: Add enum and refactor SELECT VERSION() handling - Add SelectVersionForwardingMode enum to replace magic numbers (0,1,2,3) - Refactor modes 2 and 3 to eliminate code duplication - Improve code readability and maintainability Addresses feedback from gemini-code-assist on PR #5277	5 months ago
Rene Cannao	366164ab26	Fix #5272 : Add mysql-select_version_forwarding variable for SELECT VERSION() Since ProxySQL 3.0.4, SELECT VERSION() queries were intercepted and returned ProxySQL's mysql-server_version variable instead of proxying to backends. This broke SQLAlchemy for MariaDB which expects "MariaDB" in the version string. This commit adds a new variable `mysql-select_version_forwarding` with 4 modes: - 0 = never: Always return ProxySQL's version (3.0.4+ behavior) - 1 = always: Always proxy to backend (3.0.3 behavior) - 2 = smart (fallback to 0): Try backend connection, else ProxySQL version - 3 = smart (fallback to 1): Try backend connection, else proxy (default) The implementation includes: - New global variable mysql_thread___select_version_forwarding - New function get_backend_version_for_hostgroup() to peek at backend connection versions without removing them from the pool - Modified SELECT VERSION() handler to support all 4 modes - ProxySQL backend detection to avoid recursion Mode 3 (default) ensures SQLAlchemy always gets the real MariaDB version string while maintaining fast response when connections are available.	5 months ago
Javier Jaramago Fernández	5c8a32a0b0	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-handle_unexp_ping	5 months ago
Javier Jaramago Fernández	6fea828e86	Improve logging in unexpected COM_PING packet handling Logging messages now include 'client address', 'session status' and 'data stream status'. Client address is also logged when OK packets are dispatched, this should help tracking if a client has received the expected packets or not.	5 months ago
René Cannaò	0f7ff1f374	Merge branch 'v3.0' into v3.0_pgsql-query-digest-gen-5253 Signed-off-by: René Cannaò <rene@proxysql.com>	5 months ago
Javier Jaramago Fernández	d0e88599ee	Add special handling for unexpected COM_PING packets Implements a workaround for the handling of unexpected 'COM_PING' packets received during query processing, while a resultset is yet being streamed to the client. Received 'COM_PING' packets are queued in the form of a counter. This counter is later used to sent the corresponding number of 'OK' packets to the client after 'MySQL_Session' has finished processing the current query.	5 months ago
René Cannaò	6ee087c73c	Merge pull request #5250 from sysown/v3.0-issue5248 Fix cache_empty_result=0 not caching non-empty resultsets (issue #5248)	5 months ago
Rahim Kanji	42864e8867	Improved Tokenizer for PostgreSQL - Added `process_pg_typecast()` to handle PostgreSQL type cast syntax (::) - Recognizes type casts in various contexts: 'value'::type, column::type, etc. - Added `process_array_literal()` for PostgreSQL array processing - Handles both ARRAY[] constructor and {} literal syntax - Processes multi-dimensional arrays and nested array structures - Added `process_literal_prefix_type()` for PostgreSQL prefixed literals - Processes E'' escape string constants with backslash escapes - Handles U&'' Unicode string literals with optional UESCAPE clauses - Supports x'' hex string literals and b'' bit string literals - Manages B'' bit strings and bytea literals (\\xDEADBEEF format) - Added `process_replace_boolean()` for boolean literal replacement - Replaces TRUE and FALSE literals with parameter placeholders - Maintains case-insensitive matching (true, True, TRUE, etc.) - Preserves boolean context in expressions and WHERE clauses	5 months ago
Rene Cannao	2987242d4f	Fix cache_empty_result=0 not caching non-empty resultsets (issue #5248 ) The `cache_empty_result` field in query rules has three possible values: • -1: Use global setting (`query_cache_stores_empty_result`) • 0: Do NOT cache empty resultsets, but cache non-empty resultsets • 1: Always cache resultsets (both empty and non-empty) Previously, when `cache_empty_result` was set to 0, nothing was cached at all, even for non-empty resultsets. This prevented users from disabling caching for empty resultsets while still allowing caching of non-empty resultsets on a per-rule basis. Changes: 1. Modified caching logic in MySQL_Session.cpp and PgSQL_Session.cpp to add the condition `(qpo->cache_empty_result == 0 && MyRS->num_rows)` (MySQL) and `(qpo->cache_empty_result == 0 && num_rows)` (PgSQL) to allow caching when cache_empty_result=0 AND result has rows. 2. Added comprehensive Doxygen documentation in query_processor.h explaining the semantics of cache_empty_result values. 3. Updated Query_Processor.cpp with inline comments explaining the three possible values. Now when cache_empty_result is set to 0: - Empty resultsets (0 rows) are NOT cached - Non-empty resultsets (>0 rows) ARE cached - This matches the intended per-rule behavior described in issue #5248. Fixes: https://github.com/sysown/proxysql/issues/5248	5 months ago
René Cannaò	5a314d2364	Merge pull request #4889 from sysown/v3.0_get_server_version Added handling of SELECT @@version and SELECT VERSION() without backend	5 months ago

1 2 3 4 5 ...

1091 Commits (4e39f3bdc4888ee7bd34b3fcba3b986503efe20b)