14 Commits (v3.0-5243)

Author	SHA1	Message	Date
Rene Cannao	5d786520b0	Merge remote-tracking branch 'origin/v3.0' into merge/v3.0-into-v3.0-5243 ... # Conflicts: # include/PgSQL_Logger.hpp # lib/Makefile # lib/PgSQL_Logger.cpp # lib/ProxySQL_Admin.cpp # test/tap/groups/groups.json	4 days ago
Rene Cannao	17092487cf	Add protocol-labeled logger counters and FLUSH LOGS race TAP ... - Introduce proxysql_query_logger_logged_queries_total with protocol label (mysql, pgsql) for direct query-log accounting via Prometheus. - Wire MySQL and PgSQL loggers to increment/export the counter and hook PgSQL logger metrics into admin scrape updates. - Add reg_test_5389-flush_logs_no_drop-t.cpp to stress concurrent client traffic + PROXYSQL FLUSH LOGS and assert no dropped logs via metric delta == executed queries. - Document the new TAP with doxygen comments for env knobs and flow.	5 days ago
Rene Cannao	0aa3c5e153	fix(pgsql-eventslog): harden event ownership and keep newest rows in memory table ... Problem 1: - `PgSQL_Event::set_error()` stored borrowed pointers from connection-owned strings. - This could create lifetime hazards before/around event serialization and buffering. Fix: - Add explicit ownership tracking for error fields via `free_error_on_delete`. - Make `set_error()` duplicate incoming SQLSTATE/error strings. - Free owned error strings in destructor when the event does not own all fields. - Keep copy-constructor behavior for deep-copied buffered events and disable copy-assignment to avoid accidental shallow assignment. Problem 2: - `processEvents()` could repopulate `stats_pgsql_query_events` using the beginning of the drained batch after retention pruning. - This keeps older rows instead of the most recent rows when truncation is required. Fix: - Insert the tail of the drained batch (`events.end() - numEventsToInsert`) so memory table retention always preserves latest events. - Tighten row retention arithmetic using `size_t` to avoid signed/unsigned mismatch in keep/delete calculations.	5 days ago
Rene Cannao	97e5cfee90	docs(doxygen): expand PostgreSQL logger API documentation for new advanced logging components ... Context: - The PostgreSQL advanced query logging implementation added new event, buffer, and metrics APIs. - Several newly introduced functions lacked Doxygen coverage. Changes: - Add Doxygen documentation blocks for newly introduced logging types and APIs in `include/PgSQL_Logger.hpp`: - metric enums and metric-map indexes - `PGSQL_LOG_EVENT_TYPE` - `PgSQL_Event` class and its new/updated lifecycle + serialization methods - `PgSQL_Logger_CircularBuffer` constructor/destructor and buffer operations - `PgSQL_Logger` constructor/destructor Why: - Improve maintainability and code navigation for the new PostgreSQL advanced logging feature set. - Align implementation with project expectation that introduced features include inline Doxygen documentation.	5 days ago
Rene Cannao	acbd7d0df3	Implement PostgreSQL eventslog circular buffer and sink pipeline	5 days ago
Rene Cannao	eaadabbb8e	logger: fix thread-context map race in buffered logging	5 days ago
Rene Cannao	caf324f911	logger: harden buffered logging flush/rotation semantics ... This commit addresses three correctness regressions introduced by thread-local log buffering in PR #5364 (query logging performance): 1) stale logfile pointer/UAF race during concurrent rotate/close 2) stale logfile-open state after close 3) non-global flush behavior in admin/format-switch paths What was fixed - Make `flush_and_rotate()` consume the current logfile pointer under lock - Signature changed from `std::fstream*` to `std::fstream*&` - Prevents dereferencing a stale stream pointer captured before lock acquisition while another thread rotates/closes the file - Updated declaration/definition and all call sites - Add explicit synchronization for cross-thread buffer draining - Added `LogBufferThreadContext::buffer_lock` - Any path that appends or flushes a thread buffer now locks this mutex - Guarantees force-flush from admin/config paths cannot race with worker-thread appends on the same context - Restore global forced flush semantics where required - Extended `MySQL_Logger::flush` and `PgSQL_Logger::flush` to `flush(bool force = false)` - `force=false`: preserves existing low-overhead worker-loop behavior (per-thread timeout-based flush) - `force=true`: snapshots all known thread contexts and drains both events/audit buffers regardless of timeout - `flush_log()` now calls `flush(true)` before file rotation, so admin flush and format-switch operations no longer miss pending thread buffers - Avoid unintended rotation during forced draining - In `force=true` path, flush uses `rotate_fn=nullptr` - Drains buffered payload into the current file first - `flush_log()` then performs one controlled rotate/open step - Fix logfile state tracking after close - `events_close_log_unlocked()` and `audit_close_log_unlocked()` now set `logfile_open=false` when the stream is closed - Prevents write paths from treating a closed stream as open - Remove per-thread-context dependency during metadata header write - `events_open_log_unlocked()` now uses a local `LogBuffer` for metadata emission in format=1 instead of reusing a thread context buffer - Keeps open/rotate path independent from worker context lifecycle - Keep callers consistent and non-duplicative - Since `flush_log()` now force-drains internally, removed redundant explicit `flush()` calls from: - MySQL/PgSQL `eventslog_format` switch handlers - `ProxySQL_Admin::flush_logs` Behavioral outcome - No stale stream pointer use when close/rotate interleaves with flush - No false-positive logfile-open state after close - `FLUSH LOGS` and eventslog format switch now drain all known thread buffers before rotating, preventing dropped/misplaced buffered records Validation - Built modified objects directly - Ran full debug build with GENAI enabled: make clean && export PROXYSQLGENAI=1 && make debug -j24 Build completed successfully.	5 days ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243) ... Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	2 weeks ago
Rahim Kanji	a89291c9a9	Added query logging support for PostgreSQL	5 months ago
Rahim Kanji	24fecc1f6e	Add PostgreSQL extended query (prepared statement) support in ProxySQL ... This commit lays the groundwork for handling PostgreSQL’s extended query protocol (prepared statements) by introducing parsing, it's caching and execution framework stubs. - Implement Parse handling - Implement caching of parse (reuse) - Introduce Execute framework stub for future binding and execution logic.	8 months ago
Rahim Kanji	026f458f7c	Removed last_insert_id from PostgreSQL modules	9 months ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture ... - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	1 year ago
Rahim Kanji	65fc633cf5	Generic error handling, transaction detection and handling, bug fixes ... * Implemented generic error handling to cover all the scenarios. * Extraction of results from PGconn in case of connection errors. Introduced a patch to facilitate this. * Added transaction handling. * Ready packet now includes transaction status. * Split command completion and ready packet to increase flexibility. * Resolved multiple bugs to enhance stability.	2 years ago
Rahim Kanji	e029d39007	Initial PostgreSQL commit ... * Added plain text and SCRAM authentication	2 years ago