25 Commits (feature/pgbouncer-compat)

Author	SHA1	Message	Date
Rene Cannao	17092487cf	Add protocol-labeled logger counters and FLUSH LOGS race TAP ... - Introduce proxysql_query_logger_logged_queries_total with protocol label (mysql, pgsql) for direct query-log accounting via Prometheus. - Wire MySQL and PgSQL loggers to increment/export the counter and hook PgSQL logger metrics into admin scrape updates. - Add reg_test_5389-flush_logs_no_drop-t.cpp to stress concurrent client traffic + PROXYSQL FLUSH LOGS and assert no dropped logs via metric delta == executed queries. - Document the new TAP with doxygen comments for env knobs and flow.	3 months ago
Rene Cannao	eaadabbb8e	logger: fix thread-context map race in buffered logging	3 months ago
Rene Cannao	caf324f911	logger: harden buffered logging flush/rotation semantics ... This commit addresses three correctness regressions introduced by thread-local log buffering in PR #5364 (query logging performance): 1) stale logfile pointer/UAF race during concurrent rotate/close 2) stale logfile-open state after close 3) non-global flush behavior in admin/format-switch paths What was fixed - Make `flush_and_rotate()` consume the current logfile pointer under lock - Signature changed from `std::fstream*` to `std::fstream*&` - Prevents dereferencing a stale stream pointer captured before lock acquisition while another thread rotates/closes the file - Updated declaration/definition and all call sites - Add explicit synchronization for cross-thread buffer draining - Added `LogBufferThreadContext::buffer_lock` - Any path that appends or flushes a thread buffer now locks this mutex - Guarantees force-flush from admin/config paths cannot race with worker-thread appends on the same context - Restore global forced flush semantics where required - Extended `MySQL_Logger::flush` and `PgSQL_Logger::flush` to `flush(bool force = false)` - `force=false`: preserves existing low-overhead worker-loop behavior (per-thread timeout-based flush) - `force=true`: snapshots all known thread contexts and drains both events/audit buffers regardless of timeout - `flush_log()` now calls `flush(true)` before file rotation, so admin flush and format-switch operations no longer miss pending thread buffers - Avoid unintended rotation during forced draining - In `force=true` path, flush uses `rotate_fn=nullptr` - Drains buffered payload into the current file first - `flush_log()` then performs one controlled rotate/open step - Fix logfile state tracking after close - `events_close_log_unlocked()` and `audit_close_log_unlocked()` now set `logfile_open=false` when the stream is closed - Prevents write paths from treating a closed stream as open - Remove per-thread-context dependency during metadata header write - `events_open_log_unlocked()` now uses a local `LogBuffer` for metadata emission in format=1 instead of reusing a thread context buffer - Keeps open/rotate path independent from worker context lifecycle - Keep callers consistent and non-duplicative - Since `flush_log()` now force-drains internally, removed redundant explicit `flush()` calls from: - MySQL/PgSQL `eventslog_format` switch handlers - `ProxySQL_Admin::flush_logs` Behavioral outcome - No stale stream pointer use when close/rotate interleaves with flush - No false-positive logfile-open state after close - `FLUSH LOGS` and eventslog format switch now drain all known thread buffers before rotating, preventing dropped/misplaced buffered records Validation - Built modified objects directly - Ran full debug build with GENAI enabled: make clean && export PROXYSQLGENAI=1 && make debug -j24 Build completed successfully.	3 months ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243) ... Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	3 months ago
René Cannaò	f47b0e6f6c	Changed to MySQL_Event ... Adding session Addin a placeholder for extractStmtExecuteMetadataToJson()	1 year ago
René Cannaò	cb53db81ca	Export MySQL_Logger metrics for v3.0 ... Port of #4802	1 year ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging ... See #4713 for further details	1 year ago
René Cannaò	c0745b42c2	Adding last_insert_id and last_gtid to MySQL_Logger	3 years ago
Javier Jaramago Fernández	41ef582fc5	Add new field 'stmt_id' to eventslog for 'STMT_PREPARE|STMT_EXECUTE' events	4 years ago
René Cannaò	255d9247b8	Reduce locking on MySQL_Logger writes to disk	7 years ago
René Cannaò	0831740e45	Several enhancements on Query logger ... Enhancements: - added metrics rows_affected and rows_sent - added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0 - added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871 Changing value at runtime causes the current file to be closed and a new one created - fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent Extended `eventslog_reader_sample.cpp` to support the new enhancements	7 years ago
René Cannaò	7ea33a1312	Completing initial implementation of Audit module	7 years ago
René Cannaò	9e2c1fdda1	Draft implementation of audit log	7 years ago
René Cannaò	5e05348b22	Remove spinlock from MySQL Logger #977	9 years ago
René Cannaò	abd83ff1a3	Compile 1.4.0 on on FreeBSD #898	9 years ago
René Cannaò	a6c53d0d12	Revert "Refactor header includes."	10 years ago
Maxim Bublis	30dbcc21a8	Refactor header includes. ... * reduce build time by 25%; * remove circular dependency on `cpp.h` and other headers. * fixes issue with `my_global.h` from mariadbclient which redefines `__attribute__`, that leads to broken STL and GCC intrinsics if `<algorithm>` wasn't included prior including this header; * fixes another issue with redefined `__attribute__` which leads to `_conn_exchange_t` not being aligned to cache line size; * removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize` to be `pthread_dummy(0)` which led to stack size being never adjusted; * add missing include guards to some headers; * remove unused proxysql_hash.h header.	10 years ago
René Cannaò	8e81495bee	Improved logging ... Added logging information for: - client - server - hostgroup_id - digest - duration	10 years ago
René Cannaò	31ac6625f6	Logging improvements, related to #561 ... - added new function realtime_time() to get real time - added support for multiple events, default PROXYSQL_QUERY - MySQL_Event::write() will perfom a different action depending from log_event_type - rewrote part of eventslog_reader_sample.cpp: - it had few bugs - MySQL_Event::read() will perform a different action depending from log_event_type - time are printed in real time and not monotonic time - all info about a query are printed in one line	10 years ago
René Cannaò	7830689dd9	New event logger ... Implemented new event logger that will replace protobuf It uses the same encoding of mysql protocol Added also a reader in ./tools/	11 years ago
René Cannaò	a63b7cf951	MySQL_Logger configuration ... - added global variable eventslog_filename - added global variable eventslog_filesize	11 years ago
René Cannaò	5ad97666f2	Improvement on MySQL_Logger ... added flush_log_unlocked() hardcoded max size for testing	11 years ago
René Cannaò	535030d546	MySQL_Logger starts logging using protobuf	11 years ago
René Cannaò	dbb9c111bb	Extending and testing MySQL_Logger()	11 years ago
René Cannaò	125cfa2c3b	First commit with MySQL_Logger	11 years ago