proxysql/doc/internal/pgsql_advanced_query_logging_architecture.md

PostgreSQL Advanced Query Logging Architecture

Document Status

• Status: Implemented (as-built)
• Scope: PostgreSQL advanced events logging parity with MySQL, including buffer, SQLite sinks, dump commands, scheduler sync, and metrics
• Branch: v3.0_pgsql_advanced_logging

1. Objective and Delivered Outcome

The PostgreSQL logging pipeline now supports advanced query events logging with the same operational model already used by MySQL:

• query event capture at request completion
• in-memory circular buffering
• manual dump from buffer to stats and/or stats_history
• optional periodic auto-dump to disk
• logger metrics in stats_pgsql_global and Prometheus

This implementation is additive. Existing PostgreSQL file-based events log and audit log behavior remains available.

2. As-Built Runtime Architecture

2.1 Capture path

1. PostgreSQL sessions call query logging at request completion.
2. PgSQL_Logger::log_request() builds a PgSQL_Event from session/backend state.
3. If file logging is enabled, the event is written to events log file.
4. If pgsql-eventslog_buffer_history_size > 0, the event is deep-copied and inserted into the PostgreSQL events circular buffer.

Implemented in:

• lib/PgSQL_Logger.cpp
• include/PgSQL_Logger.hpp

2.2 Buffering model

A dedicated PgSQL_Logger_CircularBuffer provides:

• thread-safe insertion/drain using mutex
• bounded size via runtime variable
• event drop accounting when the buffer is full or resized smaller

Runtime resizing is wired in PostgreSQL thread variable refresh:

• PgSQL_Thread::refresh_variables() applies eventslog_buffer_history_size changes to the live circular buffer.

Implemented in:

• include/PgSQL_Logger.hpp
• lib/PgSQL_Logger.cpp
• lib/PgSQL_Thread.cpp

2.3 Drain and persistence pipeline

PgSQL_Logger::processEvents(SQLite3DB* statsdb, SQLite3DB* statsdb_disk) drains the buffer and persists events to:

• memory table: stats_pgsql_query_events when statsdb != nullptr
• disk table: history_pgsql_query_events when statsdb_disk != nullptr

Behavior:

• batched SQLite inserts with prepared statements
• in-memory retention bound by pgsql-eventslog_table_memory_size
• query digest serialized as hex text (0x...), matching MySQL table style
• sqlstate and textual error persisted for failed queries

Implemented in:

• lib/PgSQL_Logger.cpp

3. Data Model

3.1 Memory table

stats.stats_pgsql_query_events

Columns:

• id
• thread_id
• username
• database
• start_time
• end_time
• query_digest
• query
• server
• client
• event_type
• hid
• extra_info
• affected_rows
• rows_sent
• client_stmt_name
• sqlstate
• error

Implemented in:

• include/ProxySQL_Admin_Tables_Definitions.h
• lib/Admin_Bootstrap.cpp

3.2 Disk history table

stats_history.history_pgsql_query_events

Columns match stats_pgsql_query_events.

Indexes:

• idx_history_pgsql_query_events_start_time on start_time
• idx_history_pgsql_query_events_query_digest on query_digest

Implemented in:

• include/ProxySQL_Statistics.hpp
• lib/ProxySQL_Statistics.cpp

4. Admin Interface and Control Surface

4.1 Dump commands

PostgreSQL-specific dump commands are now available:

• DUMP PGSQL EVENTSLOG FROM BUFFER TO MEMORY
• DUMP PGSQL EVENTSLOG FROM BUFFER TO DISK
• DUMP PGSQL EVENTSLOG FROM BUFFER TO BOTH

Command handling executes GloPgSQL_Logger->processEvents(...) with the selected sink targets.

These commands are exposed by the shared Admin module and are available from both Admin protocol endpoints:

• MySQL protocol on port 6032
• PostgreSQL protocol on port 6132

Implemented in:

• lib/Admin_Handler.cpp

4.2 Runtime/config variables

PostgreSQL thread variables used by advanced logging:

• pgsql-eventslog_buffer_history_size
• pgsql-eventslog_table_memory_size
• pgsql-eventslog_buffer_max_query_length

Admin scheduling variable:

• admin-stats_pgsql_eventslog_sync_buffer_to_disk

Implemented in:

• include/PgSQL_Thread.h
• include/proxysql_structs.h
• include/proxysql_admin.h
• lib/ProxySQL_Admin.cpp

5. Scheduler Integration

5.1 PostgreSQL auto-dump to disk

Admin main loop now periodically flushes PostgreSQL buffered events to history_pgsql_query_events when:

• stats_pgsql_eventslog_sync_buffer_to_disk > 0
• timer interval is elapsed

5.2 MySQL symmetry fix

The same scheduler loop now also invokes MySQL buffered events dump to disk based on stats_mysql_eventslog_sync_buffer_to_disk, ensuring symmetric behavior across both protocols.

Implemented in:

• lib/ProxySQL_Admin.cpp

6. Metrics Architecture

6.1 Logger internal metrics

PostgreSQL logger tracks:

• memory/disk copy counts
• total copy time (memory/disk)
• get-all-events calls/time/count
• total events copied (memory/disk)
• circular buffer added/dropped totals
• circular buffer current size

Implemented in:

• include/PgSQL_Logger.hpp
• lib/PgSQL_Logger.cpp

6.2 Stats table exposure

Metrics are exported to stats_pgsql_global with PgSQL_Logger_ prefix.

Implemented in:

• lib/ProxySQL_Admin_Stats.cpp

6.3 Prometheus exposure

Prometheus metric family proxysql_pgsql_logger_* is exposed through the serial metrics updater path.

Implemented in:

• lib/PgSQL_Logger.cpp
• lib/ProxySQL_Admin.cpp

7. Compatibility and Semantics

• Existing DUMP EVENTSLOG ... remains MySQL behavior for compatibility.
• New PostgreSQL syntax is explicit: DUMP PGSQL EVENTSLOG ....
• PostgreSQL event error fields use sqlstate + error (textual message).
• PostgreSQL event table uses database column naming.

8. Code Touchpoint Summary

• Logger core:
  • include/PgSQL_Logger.hpp
  • lib/PgSQL_Logger.cpp
• Thread/runtime integration:
  • lib/PgSQL_Thread.cpp
• Admin commands and scheduler:
  • lib/Admin_Handler.cpp
  • lib/ProxySQL_Admin.cpp
• Table definitions/bootstrap:
  • include/ProxySQL_Admin_Tables_Definitions.h
  • include/ProxySQL_Statistics.hpp
  • lib/Admin_Bootstrap.cpp
  • lib/ProxySQL_Statistics.cpp
• Stats/Prometheus metrics:
  • lib/ProxySQL_Admin_Stats.cpp

9. Validation and Acceptance Mapping

Implemented acceptance validation via TAP test:

• test/tap/tests/pgsql_query_logging_memory-t.cpp
• test/tap/tests/pgsql_query_logging_autodump-t.cpp

Coverage:

• table schema shape validation (stats_pgsql_query_events, history_pgsql_query_events)
• buffer dump command execution
• success/error event accounting in memory and history tables
• sqlstate capture for representative PostgreSQL errors
• non-empty textual error capture for error rows
• scheduler-driven periodic dump to history_pgsql_query_events via admin-stats_pgsql_eventslog_sync_buffer_to_disk

TAP group registration:

• test/tap/groups/groups.json (pgsql_query_logging_memory-t, pgsql_query_logging_autodump-t)