aherman
/
proxysql
mirror of https://github.com/sysown/proxysql
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document coredump_filters feature (issue #5213)

Browse Source 
Add comprehensive documentation for the on‑demand core dump generation
feature introduced in PR #4225. The documentation covers:

- coredump_filters and runtime_coredump_filters table definitions
- LOAD COREDUMP TO RUNTIME admin command and its aliases
- Configuration variables admin‑coredump_generation_threshold and
  admin‑coredump_generation_interval_ms
- Usage examples with INSERT and LOAD commands
- Important notes about case‑sensitive filenames, runtime‑only behavior,
  rate limiting, and platform support
- Troubleshooting guidance and best practices

The documentation addresses the missing documentation reported in
GitHub issue #5213.
misc251219
Rene Cannao 2 months ago
parent efe0d4fe61
commit cf8cbfd89a
1 changed files with 262 additions and 0 deletions
Show Stats Download Patch File Download Diff File

262
doc/coredump_filters_documentation.md
Unescape View File

@ -0,0 +1,262 @@
# ProxySQL: On‑Demand Core Dump Generation (coredump_filters)
## Introduction
ProxySQL includes a debugging feature that allows on‑demand generation of core dump files when specific code locations are reached. This is useful for diagnosing rare or hard‑to‑reproduce bugs without requiring a full debug build or restarting the proxy.
The feature works by:
1. **Defining filters**: Inserting file‑name and line‑number pairs into the `coredump_filters` table.
2. **Enabling filters**: Loading the filters to runtime with `LOAD COREDUMP TO RUNTIME`.
3. **Triggering core dumps**: When the macro `generate_coredump()` is executed at a filtered location, a core file is written to disk (subject to rate‑limiting and platform constraints).
Core dump generation is **rate‑limited** and **platform‑specific** (currently Linux on x86‑32, x86‑64, ARM, and MIPS architectures).
---
## Table Definitions
### `coredump_filters` (persistent configuration)
| Column | Type | Nullable | Primary Key | Description |
|----------|--------------|----------|-------------|-------------|
| filename | VARCHAR | NOT NULL | Yes | Source file name (as seen by the compiler) |
| line | INT | NOT NULL | Yes | Line number within that file |
**Primary key**: (`filename`, `line`)
**SQL definition**:
```sql
CREATE TABLE coredump_filters (
filename VARCHAR NOT NULL,
line INT NOT NULL,
PRIMARY KEY (filename, line)
);
```
### `runtime_coredump_filters` (runtime state)
This table mirrors the active filters currently loaded into memory. It is updated automatically when `LOAD COREDUMP TO RUNTIME` is executed.
**SQL definition**:
```sql
CREATE TABLE runtime_coredump_filters (
filename VARCHAR NOT NULL,
line INT NOT NULL,
PRIMARY KEY (filename, line)
);
```
---
## Configuration Variables
Two global variables control the rate‑limiting behavior:
| Variable | Default | Range | Description |
|----------|---------|-------|-------------|
| `admin‑coredump_generation_threshold` | 10 | 1–500 | Maximum number of core files that can be generated during the lifetime of the ProxySQL process. |
| `admin‑coredump_generation_interval_ms` | 30000 (30 seconds) | 0–INT_MAX | Minimum time between two consecutive core dump generations. A value of `0` disables the interval check. |
**Notes**:
- Both variables are stored in the `global_variables` table (admin database).
- Changes take effect immediately when the variable is set (no need to `LOAD … TO RUNTIME`).
- The threshold is a **global counter**; once reached, no further core dumps will be generated until the process restarts.
- The interval is measured in **milliseconds**.
---
## Admin Commands
### `LOAD COREDUMP TO RUNTIME`
Reads the `coredump_filters` table and loads the filters into memory. After this command, any location matching a filter becomes active for core dump generation.
**Aliases**:
- `LOAD COREDUMP FROM MEMORY`
- `LOAD COREDUMP FROM MEM`
- `LOAD COREDUMP TO RUN`
**Example**:
```sql
LOAD COREDUMP TO RUNTIME;
```
**Effect**:
1. Clears the previous runtime filter set.
2. Reads all rows from `coredump_filters`.
3. Converts each row into a string `"filename:line"` and stores it in an internal hash set.
4. If at least one filter exists, the global flag `coredump_enabled` is set to `true`.
### `SAVE COREDUMP` (not implemented)
As of this writing, there is **no `SAVE COREDUMP` command**. The runtime state (`runtime_coredump_filters`) is automatically updated when filters are loaded, but there is no built‑in command to persist runtime filters back to the `coredump_filters` table.
If you need to copy the active filters back to the configuration table, you can do so manually:
```sql
INSERT INTO coredump_filters SELECT * FROM runtime_coredump_filters;
```
---
## Important Notes
- **Case‑sensitive filenames**: The `filename` column must match exactly the string returned by `__FILE__` (including relative path from the source root). The comparison is case‑sensitive.
- **Runtime‑only behavior**: Filters loaded via `LOAD COREDUMP TO RUNTIME` are stored in memory only. They are lost when ProxySQL restarts. To make filters persistent, keep them in the `coredump_filters` table and reload after each restart.
- **Instance‑specific**: Filters are local to the ProxySQL instance; there is no automatic synchronization across a cluster.
- **Rate limiting**: The feature includes two safety limits (`admin‑coredump_generation_threshold` and `admin‑coredump_generation_interval_ms`) to prevent disk filling and performance degradation.
- **Platform‑specific**: Core dump generation works only on Linux x86‑32, x86‑64, ARM, and MIPS architectures. On other platforms the macro logs a warning and does nothing.
---
## Usage Example
### 1. Insert a filter
Suppose you want to generate a core dump when the function `MySQL_Data_Stream::check_data_flow()` reaches line 485 (where `generate_coredump()` is called).
First, find the exact file name as used in the source code. The macro `LOCATION()` expands to `__FILE__ ":" __LINE__`. For the file `lib/mysql_data_stream.cpp` line 485:
```sql
INSERT INTO coredump_filters (filename, line) VALUES ('lib/mysql_data_stream.cpp', 485);
```
### 2. (Optional) Adjust rate‑limiting variables
Increase the threshold and shorten the interval if you expect to trigger the dump multiple times quickly:
```sql
SET admin-coredump_generation_threshold = 50;
SET admin-coredump_generation_interval_ms = 1000; -- 1 second
```
These changes take effect immediately.
### 3. Load filters to runtime
```sql
LOAD COREDUMP TO RUNTIME;
```
### 4. Trigger the condition
Cause the code path to reach the filtered location. In this example, you would need to create a MySQL data‑stream condition where data exists at both ends of the stream (a fatal error). When that happens, ProxySQL will log:
```
[INFO] Coredump filter location 'lib/mysql_data_stream.cpp:485' was hit.
[INFO] Generating coredump file 'core.<pid>.<counter>'...
[INFO] Coredump file 'core.<pid>.<counter>' was generated ...
```
### 5. Inspect the core file
The core file is written in the current working directory of the ProxySQL process, with the name pattern `core.<pid>.<counter>` (e.g., `core.12345.0`). It is compressed using the **coredumper** library.
Analyze it with `gdb`:
```bash
gdb /usr/bin/proxysql core.12345.0
```
---
## Rate Limiting and Safety Features
To prevent disk filling and performance impact, core dump generation is protected by two mechanisms:
1. **Threshold limit**: The total number of core dumps generated during the process lifetime cannot exceed `admin‑coredump_generation_threshold` (default 10). Once the threshold is reached, `generate_coredump()` will still log the hit but will not write a new core file.
2. **Interval limit**: After a core dump is written, at least `admin‑coredump_generation_interval_ms` milliseconds must pass before another core dump can be generated (unless the interval is set to 0). This prevents burst generation when a hot code path is repeatedly executed.
Both counters are reset when `LOAD COREDUMP TO RUNTIME` is executed (or when `proxy_coredump_reset_stats()` is called internally).
---
## Platform Support
Core dump generation is **only available** on the following platforms:
- **Operating system**: Linux
- **Architectures**: x86‑32 (`__i386__`), x86‑64 (`__x86_64__`), ARM (`__ARM_ARCH_3__`), MIPS (`__mips__`)
On other platforms (e.g., FreeBSD, macOS, Windows) the `generate_coredump()` macro will log a warning and do nothing.
The feature relies on the **coredumper** library (https://github.com/elastic/coredumper), which is bundled as a dependency.
---
## Internal Implementation Details
### Macros and Functions
- `generate_coredump()`: The macro used in the source code to conditionally generate a core dump. It checks `coredump_enabled` and looks up the current `__FILE__:__LINE__` in the filter set.
- `proxy_coredump_load_filters()`: Loads a set of `"filename:line"` strings into the internal hash table.
- `proxy_coredump_generate()`: Performs the actual core dump writing, subject to rate‑limiting checks.
- `proxy_coredump_reset_stats()`: Resets the generation counter and last‑creation timestamp.
### Database‑to‑Runtime Flow
1. `LOAD COREDUMP TO RUNTIME` calls `ProxySQL_Admin::load_coredump_to_runtime()`.
2. Which calls `flush_coredump_filters_database_to_runtime()`.
3. Reads `coredump_filters` table, builds the string set, and passes it to `proxy_coredump_load_filters()`.
4. The runtime state is mirrored to `runtime_coredump_filters` via `dump_coredump_filter_values_table()`.
### Where `generate_coredump()` is Used
Currently, the macro is placed in a few strategic “fatal error” locations:
- `MySQL_Data_Stream::check_data_flow()` – when data exists at both ends of a MySQL data stream.
- `PgSQL_Data_Stream::check_data_flow()` – analogous condition for PostgreSQL.
Developers can add more `generate_coredump()` calls in other debug‑sensitive code sections.
---
## Troubleshooting
### No core file is generated even though the filter was hit
1. **Check platform support**: Verify ProxySQL is running on a supported Linux architecture.
2. **Check rate‑limiting counters**: The global threshold may have been reached. Execute `LOAD COREDUMP TO RUNTIME` to reset the counters (or restart ProxySQL).
3. **Check directory permissions**: The process must have write permission in the current working directory.
4. **Check disk space**: Ensure there is sufficient free disk space.
### Error “Coredump generation is not supported on this platform.”
The platform is not among the supported architectures. Use a different machine or consider using a debugger instead.
### Filters are not being activated after `LOAD COREDUMP TO RUNTIME`
- Verify that the `filename` matches exactly the string that `__FILE__` expands to (relative path from the source root).
- Ensure the line number is correct (check the source code for the exact line where `generate_coredump()` appears).
- Inspect the `runtime_coredump_filters` table to confirm the filters were loaded.
### High frequency of core dumps is affecting performance
Increase `admin‑coredump_generation_interval_ms` to space out the generation, or reduce `admin‑coredump_generation_threshold` to limit the total number.
---
## Best Practices
1. **Use for debugging only**: Enable coredump filters only during debugging sessions. Remove filters afterward to avoid unnecessary overhead.
2. **Limit the threshold**: Keep `admin‑coredump_generation_threshold` low (e.g., 1‑5) unless you are investigating a recurring issue.
3. **Set a reasonable interval**: A minimum interval of several seconds (e.g., 30000 ms) prevents storm generation.
4. **Document filter locations**: Keep a record of why each filter was inserted and under what condition it triggers.
5. **Monitor disk usage**: Core files can be large; ensure the working directory has enough space and consider a dedicated partition.
---
## Related Features
- **Debug filters**: ProxySQL also supports `debug_filters` for enabling debug logs at specific file‑line locations.
- **Core dump on crash**: For crash‑induced core dumps, use system‑level configuration (e.g., `ulimit -c unlimited`, `sysctl kernel.core_pattern`).
---
## Summary
The `coredump_filters` feature provides a targeted, rate‑limited way to obtain core dumps from specific code locations without restarting ProxySQL or building a debug binary. It is a valuable tool for diagnosing elusive bugs in production‑like environments.
Remember that core dump generation is **platform‑dependent** and **rate‑limited**; always verify support and adjust the configuration variables according to your debugging needs.
Write Preview
Loading…
Cancel
Save