5 Commits (feature/pgsql-native-backend-protocol)

Author	SHA1	Message	Date
Rene Cannao	9ea2b31bf5	fix(mysqlx): match upstream X-Protocol error code for compression-without-algorithm ... Closes #5696 (P2 — MySQL Router parity gap, compression error code). Before this change the mysqlx plugin emitted error code 5008 for every compression-related failure ("Compression frame received without negotiated algorithm", parse failures, decompression failures, etc.). That code does not match upstream MySQL X plugin: 5008 in plugin/x/src/xpl_error.h is ER_X_BAD_CONNECTION_SESSION_ATTRIBUTE_TYPE — a totally unrelated session-attribute error. A client written specifically against the upstream X-Protocol error contract would see the wrong error class entirely. Authoritative source for the canonical codes: plugin/x/src/xpl_error.h in the MySQL 8.4 source tree (deps/mysql-connector-c-8.4.0/.../xpl_error.h in this repo). Mapping: - 5170 ER_X_FRAME_COMPRESSION_DISABLED — compression frame received when the client did not negotiate compression. Upstream message: "Client didn't enable the compression." Reproduced verbatim. - 5174 ER_X_BAD_COMPRESSED_FRAME — structural problem with the Compression envelope (empty body, malformed protobuf, bogus uncompressed_size hint, decompressed payload that doesn't reframe into valid X messages). - 5171 ER_X_DECOMPRESSION_FAILED — algorithm-side failure (lz4/zstd error, OOM, stall, decompressed payload exceeds cap). - 5000 ER_X_BAD_MESSAGE — server-direction compression on the client→server path; treated as a wrong-direction message rather than a compression-specific failure. The pre-auth capability replay-cap error (also previously emitting 5008) is a ProxySQL-specific guardrail with no upstream analogue and is left untouched in this commit — the parity fix is scoped to the genuine compression error sites. Test updates: - mysqlx_message_dispatch_unit-t::test_dispatch_compression_rejected: asserts 5170 (was 5008). - mysqlx_compression_unit-t: oversize/garbage rejections assert 5171, no-negotiation rejection asserts 5170, "no error frame" tests accept any of 5170/5171/5174 (renamed got_5008 → got_compression_err). - test_compression_without_negotiation_still_5008 → test_compression_without_negotiation_still_5170. Build: NOJEMALLOC=1 WITHASAN=1 PROXYSQLGENAI=1 make build_lib + plugin .so + build_tap_test_debug all green. Tests: mysqlx_session_unit-t (87/87), mysqlx_message_dispatch_unit-t (98/98), mysqlx_compression_unit-t (64/64), mysqlx_tls_unit-t (18/18), mysqlx_thread_unit-t (25/25), mysqlx_concurrent_unit-t (6/6) — all ASAN-clean.	2 months ago
Rene Cannao	e68f72cf99	test(mysqlx): add diag logging + fix broken plan/schema in unit tests ... Two related changes to all mysqlx_*_unit-t.cpp files. # 1. Diagnostic logging across all 21 mysqlx unit tests Before this commit none of the mysqlx unit tests emitted any output beyond raw TAP `ok N`/`not ok N` lines. When CI-unit-tests-asan- coverage hung mid-run on plugin-chassis (run 25021206900), the per- test log file showed `ok 1 ... ok 6` and then nothing for ~110 minutes until the workflow's 120-minute timeout killed the job. We could not even tell *which* test was hung — the bash loop in the workflow only logs `PASS:`/`FAIL:` after a binary exits. Each binary now does: - `setvbuf(stdout, nullptr, _IOLBF, 0)` as the first statement of main(). Without this, when CI redirects stdout to a file the default block-buffered mode swallows any output that hasn't filled a 4KB-ish buffer — including diag lines emitted just before a blocking syscall. Line-buffering means each `\n`-terminated diag is flushed before the syscall, so a hang has 1-line-resolution attribution in the log. - `diag("=== <filename> starting ===")` right after `plan(...)`, so the log always has at least one line proving the binary loaded and reached main() (rules out crashes during static init). - `diag(">>> %s", __func__)` at the top of every `static void test_*()` subtest function called from main(), so a hang or assertion failure points at the responsible subtest. - In `mysqlx_concurrent_unit-t`, additional checkpoint diags inside `test_concurrent_handshakes` ("starting Mysqlx_Thread", "spawning N clients", "all clients spawned, joining", ..., "thr.stop() returned") since the surrounding test was the one that actually hung in CI; this gives a 7-checkpoint resolution within the body. This change is the reason we were able to bisect the listener O_NONBLOCK bug fixed in the previous commit: with these diags the hang point ("all clients spawned, joining" → never reached "all clients joined, sleeping 500ms") was visible in the per-test log within seconds of running it. # 2. Plan/fixture corrections in 4 tests that never actually passed These tests were green in CI by virtue of nothing in the workflow checking their exit codes (CI-mysqlx exercised a different path); the moment CI-unit-tests-asan-coverage started actually running them they came back red. Each was a real bug in the test harness, not in the production code: - mysqlx_backend_auth_unit-t.cpp: plan(42) but only 34 ok() can fire on the success path. State_transitions emits 23, the remaining six functions emit 11 — total 34. plan(42) was never reachable; fixed to plan(34). - mysqlx_session_unit-t.cpp: plan(62) but only 60 fire. Four `ok( false, ...)` lines live in protobuf-parse failure branches that the success path skips; fixed to plan(60). - mysqlx_config_store_unit-t.cpp: load_from_runtime() SELECTs from runtime_mysqlx_variables (the variables table the production code uses to pick up thread_pool_size / connect_timeout / tls_mode / max_cached_connections at runtime), but the test's fixture only created 3 of the 4 mysqlx tables. Added kRuntimeMysqlxVariablesDdl and the matching execute() — same shape the sibling mysqlx_config_store_concurrent_unit-t already uses. - mysqlx_route_store_unit-t.cpp: ad-hoc minimal DDL for runtime_mysql_users / runtime_mysql_servers omitted columns the production query SELECTs (password, weight). Switched to the canonical ADMIN_SQLITE_RUNTIME_MYSQL_USERS and ADMIN_SQLITE_TABLE_RUNTIME_MYSQL_SERVERS macros and added the runtime_mysqlx_variables table. After this commit `for t in mysqlx_*_unit-t; do ./$t; done` is 21/21 green (754 ok asserts total, 0 not_ok, 0 timeouts), 10/10 stable across repeated runs of the previously-flaky mysqlx_concurrent_unit-t.	2 months ago
Rene Cannao	aef01ef0be	feat(mysqlx): compress outbound server frames (Phase 3) ... Phase 3 of three-phase X Protocol compression support: server→client compression on outbound frames. Wraps up the MVP — CapabilitiesSet negotiation (Phase 1), inbound decompression (Phase 2), and now outbound compression all roundtrip. What this commit does ===================== forward_frame_to_client() — the chokepoint for backend→client frame forwarding — now routes through send_to_client_compressed(), which: 1. Returns early to plain client_ds_.enqueue_frame() when compression is not negotiated OR the body is below COMPRESSION_MIN_OUTPUT_BYTES (50 bytes — same threshold the upstream MySQL X plugin uses; per- message envelope + framing overhead would otherwise dwarf savings). 2. With combine_mixed_messages = false: emits one Compression frame per body. The protobuf has server_messages set to the original frame's msg_type and uncompressed_size set to the original body length, so a spec-compliant client can decompress in place. 3. With combine_mixed_messages = true: appends a fully-framed copy of the body to compress_batch_framed_ and bumps compress_batch_count_. Once the count reaches max_combine_messages (defaulting to 64 if the client didn't supply one), or flush_compression_batch() is invoked at a natural boundary, all buffered frames are emitted as one Compression message with NEITHER server_messages nor client_messages set — payload is the concatenated stream of framed X messages, matching the spec's third payload shape. The flush points are: - handler_waiting_server_msg() when it sees a terminal frame (end of a result set / final OK / etc.) — the response is "done" so anything still buffered must reach the wire before we go back to WAITING_CLIENT_XMSG. - handler_session_reset_waiting() on the ERROR path before write_to_net(), same reasoning. Mid-response we deliberately do NOT flush — that would defeat the combine_mixed_messages benefit by emitting a Compression message per batch hit, which is exactly what we're trying to avoid for streaming result sets. The count cap bounds how long a single batch can grow. Compressor selection mirrors Phase 2: - zstd_stream uses ZSTD_compressCCtx() with a per-session ZSTD_CCtx that's lazily allocated and freed in reset_compression_state(). - lz4_message uses LZ4_compress_default() one-shot. On compressor failure (ZSTD_isError or lz4 returning <= 0) the helper falls back to enqueueing the body uncompressed — losing compression benefit beats dropping the message. The session itself stays healthy. Tests ===== mysqlx_compression_unit-t now plans 64 sub-tests, all passing: - Phase 1: 22 sub-tests for capability negotiation - Phase 2: 17 sub-tests for inbound decompression - Phase 3 (new, 25 sub-tests): - zstd round-trip: frame on the wire is COMPRESSION with server_messages = NOTICE; payload decompresses back to the exact original 200-byte body - lz4 round-trip, same shape - below-threshold passthrough: 20-byte body emitted as plain NOTICE, not COMPRESSION (verifies the fast path) - combine_mixed_messages with max_combine_messages=3: three successive sends produce ONE Compression frame whose decompressed payload contains all three NOTICE bodies in order, with neither server_messages nor client_messages set - compression-disabled passthrough: when no negotiation, the helper acts as a plain enqueue_frame() (proves a client that never opts in is unaffected) A few new test-only accessors expose the batch state and send_to_client_compressed() entry point so tests can exercise the output path without wiring a fake backend (which would block on connect() in the dispatcher anyway). Verification ============ Top-level `PROXYSQLGENAI=1 make` builds cleanly. mysqlx_compression_unit-t passes 64/64. plugin_lifecycle_unit-t (26/26) passes unchanged. mysqlx_session_unit-t still has its two pre-existing failures at sub-tests 33-34 (auth flow), unrelated to compression and present on this branch before the Phase 1 commit.	2 months ago
Rene Cannao	b1fd6b31fc	feat(mysqlx): decompress incoming Compression messages (Phase 2) ... Phase 2 of three-phase X Protocol compression support: client→server decompression. Compression on the server→client path (Phase 3) still goes out uncompressed; CapabilitiesSet stores the negotiation in Phase 1 and this commit consumes it. What this commit does ===================== When the client sends a Mysqlx.Connection.Compression message AND compression has been negotiated, we now: 1. Parse the Compression protobuf — payload bytes, optional uncompressed_size hint, optional client_messages tag. 2. Decompress the payload using the negotiated algorithm: - zstd_stream: streaming decompression via a ZSTD_DCtx that persists across messages on the same session (per spec — successive Compression frames may continue a single zstd stream, so we cannot recreate the context per frame). - lz4_message: one-shot LZ4_decompress_safe. The X spec defines lz4_message as one independent frame per message, so no persistent context is needed. 3. Feed the decompressed bytes back into client_ds_.feed_bytes() so the existing frame parser picks them up. Two payload shapes per spec are handled: - client_messages set: payload is one decompressed body of that type — we re-frame with a 5-byte X header in front. - neither set: payload is already a sequence of fully-framed X messages — fed verbatim. 4. The dispatch loop re-enters on the same handler tick (to_process set), so a Compression-wrapped StmtExecute runs end-to-end without an extra network roundtrip. If client_messages is unset and server_messages IS set, the message is rejected (5008): server_messages on the c→s path is always wrong. Anti-bomb / bounds ================== COMPRESSION_MAX_DECOMPRESSED_BYTES = 16 MiB caps how much output a single Compression message can produce, mirroring MysqlxDataStream's existing on-the-wire X_MAX_PAYLOAD_SIZE so a Compression frame never expands beyond what the rest of the data plane can handle anyway. If the client provides an uncompressed_size hint smaller than 16 MiB we honor that as the tighter bound. Hint of 0 with non-empty payload is treated as malformed. For zstd, the decompression loop re-checks the cap before each ZSTD_decompressStream call and bails (with 5008) the moment the output buffer would exceed it. A no-progress condition (zout.pos == 0 with input remaining) also bails — that catches malformed streams that would otherwise spin forever. For lz4, LZ4_decompress_safe(dstCapacity = cap) naturally enforces the limit: the decompressor refuses to write past dstCapacity and returns an error code we map to 5008. Linkage / build =============== The plugin .so is dlopen'd with RTLD_LOCAL, so symbols from libzstd / liblz4 that the proxysql binary may already pull in are NOT visible to the plugin. This commit links the static deps/zstd/zstd/lib/libzstd.a + deps/lz4/lz4/lib/liblz4.a archives directly into the .so so the plugin is self-contained. The session header forward-declares ZSTD_DCtx / ZSTD_CCtx so the zstd headers don't leak through include/mysqlx_session.h into other translation units. Tests ===== mysqlx_compression_unit-t now covers Phase 2 end-to-end (39 sub- tests total, all passing): - Phase 1: capability negotiation (22 sub-tests, unchanged) - decompress zstd_stream client_messages=SQL_STMT_EXECUTE, inner StmtExecute reaches dispatch (no 5008, session moves past WAITING_CLIENT_XMSG) - same for lz4_message - oversize bomb attempt: lie about uncompressed_size, send a 1 MiB-of-zeros payload — rejected with 5008 - garbage compressed payload: rejected with 5008 - sanity: COMPRESSION before negotiation still 5008 Verification ============ Top-level `PROXYSQLGENAI=1 make` builds cleanly; the plugin .so links against the static zstd + lz4 archives. plugin_lifecycle_unit-t (26/26) passes unchanged. mysqlx_session_unit-t still has its two pre-existing failures at sub-tests 33-34, identical to behavior before this commit (verified by stashing only the Phase 2 edits to plugins/mysqlx/{src,include}/* + Makefile and rebuilding). The mysqlx_thread_unit-t / mysqlx_concurrent_unit-t hangs are also pre-existing and reproduce on the unmodified branch.	2 months ago
Rene Cannao	f19be5f3a0	feat(mysqlx): negotiate X Protocol compression capability (Phase 1) ... Phase 1 of three-phase X Protocol compression support: capability negotiation only. The COMPRESSION message itself is still rejected with error 5008 in dispatch_client_message(); Phase 2 wires up decompression on input and Phase 3 wires up compression on output. What this commit does ===================== - send_capabilities() now advertises a `compression` capability listing the algorithms the plugin can support: zstd_stream and lz4_message. Both libraries (libzstd, liblz4) are already statically linked into libproxysql.a / pulled into the unit-test link line, so this does not introduce any new runtime dependency. - handler_capabilities_set() detects when a client sets the `compression` capability and parses the OBJECT value's sub-keys: - `algorithm` (required string) — must match an advertised value; anything else is rejected with X-Protocol error 5052 (the capability-prepare-failed code per the spec). - `server_combine_mixed_messages` / `combine_mixed_messages` (bool) - `server_max_combine_messages` / `max_combine_messages` (uint) Both spelling variants are accepted because mysql-connector-python emits the short form while libmysqlclient emits the spec form, and the upstream MySQL server tolerates both. - Negotiation outcome is stored on MysqlxSession via three new members (compression_algo_, compression_combine_mixed_messages_, compression_max_combine_messages_). Phase 2 / Phase 3 will read these to drive (de)compression. They are reset by both init() and reset() so a session reuse does not inherit stale negotiation. - Unsupported algorithms (e.g. deflate_stream) and structurally malformed values (wrong protobuf type, missing algorithm field) are rejected with a non-fatal X-Protocol Error frame — the session remains healthy so the client can either retry CapabilitiesSet or proceed without compression. Why 5052 (non-fatal) and not 5008 --------------------------------- 5008 is the runtime "compression message arrived but compression is disabled" error and stays in dispatch_client_message() for now. 5052 is the spec-defined "capability prepare failed" status used during the CapabilitiesSet handshake; treating an unknown algorithm as a capability error matches the upstream MySQL X plugin's behavior and lets compliant clients downgrade to no-compression on the same connection. Tests ===== New unit test: test/tap/tests/unit/mysqlx_compression_unit-t.cpp (22 sub-tests, all passing). Covers: - CapGet response advertises `compression` with both algorithms - CapSet zstd_stream + combine hints accepted, stored on session - CapSet lz4_message accepted, stored on session - CapSet deflate_stream rejected with 5052 (non-fatal), session remains healthy with no algorithm set - CapSet with wrong-shape compression value (scalar instead of object) rejected with 5052 Existing mysqlx_session_unit-t / plugin_lifecycle_unit-t still pass (the two pre-existing failures in mysqlx_session_unit-t at sub-tests 33-34, "auth succeeded for correct password" and "session in WAITING_CLIENT_XMSG after auth", are present on this branch before this commit and unrelated to compression — verified by stashing only the mysqlx_session.{cpp,h} edits and re-running). Build infra fix piggy-backed on this commit ------------------------------------------- test/tap/tests/unit/Makefile referenced $(SQLITE3_LDIR)/../libsqlite_rembed.a in the GenAI-tier link line, but that .a is no longer produced on this branch (it was the artifact of the now-removed sqlite-rembed Rust extension). Without removing this stale reference, no plugin/mysqlx unit test can link, which blocks verifying the new compression test alongside the existing mysqlx tests required by the task. Mirrors the upstream fix that already landed on sibling branches as commit 44a6c9d58 ("build(test): drop libsqlite_rembed.a from TAP and rag test link order"). Phase 2 (decompression) and Phase 3 (compression on output) will follow as separate commits; the COMPRESSION dispatch handler still emits 5008 until Phase 2 lands.	2 months ago