proxysql/doc/internal/MySQL_Logging_Prepared_Statements_Parameters.md

ProxySQL Prepared Statement Parameter Tracking Reference Manual

This document describes the changes made to the logging of prepared statement parameters in ProxySQL.
Two logging formats are supported:

• Binary Format (Format 1) via write_query_format_1()
• JSON Format (Format 2) via write_query_format_2_json() (also using extractStmtExecuteMetadataToJson())

1. Overview

ProxySQL logs query events in two different formats.
For prepared statements (i.e. COM_STMT_EXECUTE events), the logging functions now log extra parameter details.
Both formats encode the number of parameters and, for each parameter, log its type and a human-readable string value, with special handling for NULL values.

2. Binary Format Details (write_query_format_1())

When logging a prepared statement execution in binary format, the following steps occur:

• Encoded Basic Query Data: Standard fields such as event type, thread ID, username, schema name, client/server info, timestamps, query digest, and the query text itself are encoded using a variable-length encoding scheme.

• Logging Prepared Statement Parameters: If the event type is PROXYSQL_COM_STMT_EXECUTE and valid statement metadata is present:

• The number of parameters is encoded using mysql_encode_length().

• A null bitmap is constructed with one bit per parameter (set when the parameter is NULL).

• For each parameter:

  • Two bytes are written to store the parameter's type.
  • If the parameter is not NULL, the function getValueForBind() is used to convert the binary data into a human-readable string.
  • The length of the converted value is encoded (again using mysql_encode_length()), and then the raw bytes of the converted value are written.

• Sequential Write Process: All the fields are written sequentially. First, the total length of the event record is written (as a fixed-size 8-byte field), then each encoded field and the additional prepared statement parameter details.

3. JSON Format Details (write_query_format_2_json())

In JSON format, the logging function produces a JSON object that includes all standard query details plus additional fields for prepared statement parameters:

• Base JSON Object Information: The JSON object includes key fields such as thread_id, username, schemaname, client, server information (if available), timestamps (starttime, endtime), duration, digest, etc.

• Event Type: Depending on the event type, the field "event" is set to "COM_STMT_EXECUTE" (or "COM_STMT_PREPARE" / "COM_QUERY") accordingly.

• Logging Prepared Statement Parameters: If the event type is PROXYSQL_COM_STMT_EXECUTE and the session contains valid statement metadata:

• The helper function extractStmtExecuteMetadataToJson(json &j) is called.

• This function iterates over each parameter:

  • For NULL parameters, it logs "type": "NULL" and "value": null.
  • For non-NULL parameters, it uses getValueForBind() to obtain the parameter type (e.g., "INT24", "VARCHAR") and its string representation.

• The result is a JSON array attached to the key "parameters", where each element is an object with keys "type" and "value".

• Output: Finally the JSON object is dumped as a single line in the log file using j.dump().

4. Detailed Example

For example, consider a prepared statement with two parameters. The resulting JSON log might look like:

{
    "hostgroup_id": 3,
    "thread_id": 12345,
    "event": "COM_STMT_EXECUTE",
    "username": "dbuser",
    "schemaname": "production",
    "client": "192.0.2.1",
    "server": "10.0.0.5",
    "rows_affected": 1,
    "query": "INSERT INTO foo (bar1, bar2) VALUES (?, ?)",
    "starttime_timestamp_us": 1617181920000000,
    "starttime": "2021-03-31 12:32:00.000000"
    "endtime_timestamp_us": 1617181921000000,
    "endtime": "2021-03-31 12:32:01.000000",
    "duration_us": 1000000,
    "digest": "0x0000000000000001", "client_stmt_id": 101,
    "parameters": [
        {
            "type": "INT24",
            "value": "42"
        },
        {
            "type": "VARCHAR",
            "value": "example value"
        }
    ]
}

This JSON structure clearly shows each parameter’s type and value, which aids in debugging and analysis.

5. Usage Considerations

• Debuggability: The enhanced logging allows administrators to see the full details of the parameters that are bound to a prepared statement. This is crucial for troubleshooting and performance analysis.
• Variable-Length Encoding: Both binary and JSON formats rely on helper functions (mysql_encode_length() and getValueForBind()) to ensure that variable-length integers and parameter values are handled efficiently and unambiguously.
• Thread Safety and Performance: The logging functions acquire write locks just before writing to disk to minimize contention. The design ensures that logging does not adversely impact performance while retaining detailed information.

6. Summary

The changes in both write_query_format_1() and write_query_format_2_json() provide a comprehensive way to log prepared statement parameters:

• In binary format, parameters are logged with a parameter count, a null bitmap, a two-byte parameter type, and then a variable-length encoded value.
• In JSON format, parameters appear as an array of objects under the "parameters" key, with each object containing "type" and "value" fields.

These enhancements aim to improve troubleshooting capabilities and ensure that all necessary information is captured for later analysis.