proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	5d786520b0	Merge remote-tracking branch 'origin/v3.0' into merge/v3.0-into-v3.0-5243 # Conflicts: # include/PgSQL_Logger.hpp # lib/Makefile # lib/PgSQL_Logger.cpp # lib/ProxySQL_Admin.cpp # test/tap/groups/groups.json	4 weeks ago
Rene Cannao	82a28a2e62	fix(pgsql-eventslog): close remaining minor review findings in logger internals Address two minor correctness/style follow-ups from final review: 1) `PgSQL_Event::set_error()` overwrite safety - Free previous `sqlstate` / `errmsg` buffers when either ownership mode is active (`free_on_delete` or `free_error_on_delete`) before replacing pointers. - This avoids latent leaks if `set_error()` is called on deep-copied events. 2) Explicit atomic access for circular-buffer capacity - Keep `buffer_size` atomic and use explicit `load/store` operations with `std::memory_order_relaxed` in hot-path and helper methods. - Update log-path check, insert path, getter, and setter to use explicit atomic operations for clearer thread-safety semantics.	4 weeks ago
Rene Cannao	17092487cf	Add protocol-labeled logger counters and FLUSH LOGS race TAP - Introduce proxysql_query_logger_logged_queries_total with protocol label (mysql, pgsql) for direct query-log accounting via Prometheus. - Wire MySQL and PgSQL loggers to increment/export the counter and hook PgSQL logger metrics into admin scrape updates. - Add reg_test_5389-flush_logs_no_drop-t.cpp to stress concurrent client traffic + PROXYSQL FLUSH LOGS and assert no dropped logs via metric delta == executed queries. - Document the new TAP with doxygen comments for env knobs and flow.	4 weeks ago
Rene Cannao	0aa3c5e153	fix(pgsql-eventslog): harden event ownership and keep newest rows in memory table Problem 1: - `PgSQL_Event::set_error()` stored borrowed pointers from connection-owned strings. - This could create lifetime hazards before/around event serialization and buffering. Fix: - Add explicit ownership tracking for error fields via `free_error_on_delete`. - Make `set_error()` duplicate incoming SQLSTATE/error strings. - Free owned error strings in destructor when the event does not own all fields. - Keep copy-constructor behavior for deep-copied buffered events and disable copy-assignment to avoid accidental shallow assignment. Problem 2: - `processEvents()` could repopulate `stats_pgsql_query_events` using the beginning of the drained batch after retention pruning. - This keeps older rows instead of the most recent rows when truncation is required. Fix: - Insert the tail of the drained batch (`events.end() - numEventsToInsert`) so memory table retention always preserves latest events. - Tighten row retention arithmetic using `size_t` to avoid signed/unsigned mismatch in keep/delete calculations.	4 weeks ago
Rene Cannao	c5191d21f8	logger: avoid log drops during rotate close/open window	4 weeks ago
Rene Cannao	eba795d729	fix(pgsql-eventslog): make dump-to-memory sizing protocol-agnostic across admin ports Context: - Admin commands are exposed by one shared Admin module over two protocols: - MySQL admin endpoint (`6032`) - PostgreSQL admin endpoint (`6132`) - `DUMP PGSQL EVENTSLOG ...` must behave identically regardless of which admin protocol executes the command. Issue fixed: - PGSQL events dump path used thread-local PGSQL sizing variables when draining to memory (`stats_pgsql_query_events`). - When command execution happened from a non-PGSQL protocol admin worker, TLS values could be out of sync for PGSQL sizing decisions. Changes: - In `lib/PgSQL_Logger.cpp`, add runtime-size helper functions that read canonical values from `GloPTH->variables` with TLS fallback: - `eventslog_buffer_max_query_length` - `eventslog_table_memory_size` - Use these helpers in: - `PgSQL_Event` copy constructor (buffered query truncation) - `PgSQL_Logger::processEvents()` (in-memory retention for `stats_pgsql_query_events`) - Update internal architecture doc to explicitly state command availability from both admin protocol endpoints. Why: - Ensure `DUMP PGSQL EVENTSLOG ...` semantics remain consistent across `6032` and `6132`, matching the shared-admin-module design.	4 weeks ago
Rene Cannao	acbd7d0df3	Implement PostgreSQL eventslog circular buffer and sink pipeline	4 weeks ago
Rene Cannao	caf324f911	logger: harden buffered logging flush/rotation semantics This commit addresses three correctness regressions introduced by thread-local log buffering in PR #5364 (query logging performance): 1) stale logfile pointer/UAF race during concurrent rotate/close 2) stale logfile-open state after close 3) non-global flush behavior in admin/format-switch paths What was fixed - Make `flush_and_rotate()` consume the current logfile pointer under lock - Signature changed from `std::fstream` to `std::fstream&` - Prevents dereferencing a stale stream pointer captured before lock acquisition while another thread rotates/closes the file - Updated declaration/definition and all call sites - Add explicit synchronization for cross-thread buffer draining - Added `LogBufferThreadContext::buffer_lock` - Any path that appends or flushes a thread buffer now locks this mutex - Guarantees force-flush from admin/config paths cannot race with worker-thread appends on the same context - Restore global forced flush semantics where required - Extended `MySQL_Logger::flush` and `PgSQL_Logger::flush` to `flush(bool force = false)` - `force=false`: preserves existing low-overhead worker-loop behavior (per-thread timeout-based flush) - `force=true`: snapshots all known thread contexts and drains both events/audit buffers regardless of timeout - `flush_log()` now calls `flush(true)` before file rotation, so admin flush and format-switch operations no longer miss pending thread buffers - Avoid unintended rotation during forced draining - In `force=true` path, flush uses `rotate_fn=nullptr` - Drains buffered payload into the current file first - `flush_log()` then performs one controlled rotate/open step - Fix logfile state tracking after close - `events_close_log_unlocked()` and `audit_close_log_unlocked()` now set `logfile_open=false` when the stream is closed - Prevents write paths from treating a closed stream as open - Remove per-thread-context dependency during metadata header write - `events_open_log_unlocked()` now uses a local `LogBuffer` for metadata emission in format=1 instead of reusing a thread context buffer - Keeps open/rotate path independent from worker context lifecycle - Keep callers consistent and non-duplicative - Since `flush_log()` now force-drains internally, removed redundant explicit `flush()` calls from: - MySQL/PgSQL `eventslog_format` switch handlers - `ProxySQL_Admin::flush_logs` Behavioral outcome - No stale stream pointer use when close/rotate interleaves with flush - No false-positive logfile-open state after close - `FLUSH LOGS` and eventslog format switch now drain all known thread buffers before rotating, preventing dropped/misplaced buffered records Validation - Built modified objects directly - Ran full debug build with GENAI enabled: make clean && export PROXYSQLGENAI=1 && make debug -j24 Build completed successfully.	4 weeks ago
vramesha	71143c2e84	Address AI review comments - Added <pthread.h> to log_utils.h and removed the header from log_utils.cpp - Pass 0 to reset_time argument in flush_and_rotate inside logger's destructor	1 month ago
vramesha	091942e9d6	Address AI review suggestions	1 month ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243 ) Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	1 month ago
Rahim Kanji	2702c2092f	Fixed typo	6 months ago
Rahim Kanji	9b0366a89a	Improved PostgreSQL Logger	6 months ago
Rahim Kanji	a89291c9a9	Added query logging support for PostgreSQL	6 months ago
Rahim Kanji	bfc1484923	Added SET statement tracking tests	8 months ago
Rahim Kanji	02eb58db79	Added SET statement tracking support	8 months ago
Rahim Kanji	48ba491cb7	Add support for unnamed Describe Portal and improve extended query handling - Always send Describe Portal to the backend - Enforce check for unnamed portals in Bind, Describe, Close, and Execute; raise errors for named portals - Improve error handling for extended query flow - Introduce Extended_Query_Info struct to unify extended query-related members and reduce redundancy	8 months ago
Rahim Kanji	d5fc1fefff	Renamed pending_query to extended_query_frame Improved extended query frame handling	9 months ago
Rahim Kanji	24fecc1f6e	Add PostgreSQL extended query (prepared statement) support in ProxySQL This commit lays the groundwork for handling PostgreSQL’s extended query protocol (prepared statements) by introducing parsing, it's caching and execution framework stubs. - Implement Parse handling - Implement caching of parse (reuse) - Introduce Execute framework stub for future binding and execution logic.	9 months ago
Rahim Kanji	026f458f7c	Removed last_insert_id from PostgreSQL modules	9 months ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	2 years ago
Rahim Kanji	d01e883638	Renamed 'schemaname' to 'dbname' in PgSQL_Connection_userinfo	2 years ago
René Cannaò	fde8ace076	Removed json.hpp from include files This seems to speedup compile time, and also lead to smaller binary	2 years ago
Rahim Kanji	e029d39007	Initial PostgreSQL commit * Added plain text and SCRAM authentication	2 years ago

24 Commits (70bf12735eb7fe3a5f5f6ba2df774ecc497ea9ef)