proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	c5a7fc31f7	Add external LLM setup guide and live testing script - test_external_live.sh: Interactive script for testing with custom LLM - EXTERNAL_LLM_SETUP.md: Complete guide for external model configuration Covers: - Custom LLM endpoint configuration for NL2SQL - llama-server configuration for embeddings - Architecture overview - Configuration variables - Testing procedures - Troubleshooting tips	1 month ago
Rene Cannao	3b7033f44d	Add vector features verification script Simple script to verify that all vector features are properly implemented: - NL2SQL vector cache methods - Anomaly threat pattern management - sqlite-vec integration - GenAI module integration - Documentation completeness	1 month ago
Rene Cannao	f5c18fd8d7	scripts: Add threat pattern documentation script Add helper script showing sample threat patterns that can be added to the Anomaly Detection system for testing embedding similarity. Includes 10 sample patterns: 1. OR 1=1 tautology (severity 9) 2. UNION SELECT data extraction (severity 8) 3. Comment injection (severity 7) 4. Sleep-based DoS (severity 6) 5. Benchmark-based DoS (severity 6) 6. INTO OUTFILE exfiltration (severity 9) 7. DROP TABLE destruction (severity 10) 8. Schema probing (severity 3) 9. CONCAT injection (severity 8) 10. Hex encoding bypass (severity 7)	1 month ago
Rene Cannao	3f44229e28	feat: Add MCP AI Tool Handler for NL2SQL with test script Phase 5: MCP Tool Implementation for NL2SQL This commit implements the AI Tool Handler for the MCP (Model Context Protocol) server, exposing NL2SQL functionality as an MCP tool. New Files: - include/AI_Tool_Handler.h: Header for AI_Tool_Handler class - Provides ai_nl2sql_convert tool via MCP protocol - Wraps NL2SQL_Converter and Anomaly_Detector - Inherits from MCP_Tool_Handler base class - lib/AI_Tool_Handler.cpp: Implementation - Implements ai_nl2sql_convert tool execution - Accepts parameters: natural_language (required), schema, context_tables, max_latency_ms, allow_cache - Returns JSON response with sql_query, confidence, explanation, cached, cache_id - scripts/mcp/test_nl2sql_tools.sh: Test script for NL2SQL MCP tool - Tests ai_nl2sql_convert via JSON-RPC over HTTPS - 10 test cases covering SELECT, WHERE, JOIN, aggregation, etc. - Includes error handling test for empty queries - Supports --verbose, --quiet options Modified Files: - include/MCP_Thread.h: Add AI_Tool_Handler forward declaration and pointer - lib/Makefile: Add AI_Tool_Handler.oo to _OBJ_CXX list - lib/ProxySQL_MCP_Server.cpp: Initialize and register AI tool handler - Creates AI_Tool_Handler with GloAI components - Registers /mcp/ai endpoint - Adds cleanup in destructor MCP Tool Details: - Endpoint: /mcp/ai - Tool: ai_nl2sql_convert - Parameters: - natural_language (string, required): Natural language query - schema (string, optional): Database schema name - context_tables (string, optional): Comma-separated table list - max_latency_ms (integer, optional): Max acceptable latency - allow_cache (boolean, optional): Check semantic cache (default: true) Testing: Run the test script with: ./scripts/mcp/test_nl2sql_tools.sh [--verbose] [--quiet] See scripts/mcp/test_nl2sql_tools.sh --help for usage. Related: Phase 1-4 (Documentation, Unit Tests, Integration Tests, E2E Tests) Related: Phase 6-8 (User Docs, Developer Docs, Test Docs)	1 month ago
Rene Cannao	aee9c3117b	test: Add E2E test script for NL2SQL - test_nl2sql_e2e.sh: End-to-end testing with --mock and --live modes - Tests complete workflow from natural language to executed SQL - Includes test schema setup, LLM configuration, and 8 test cases - Supports both mocked LLM responses (fast) and live LLM testing	1 month ago
Rene Cannao	fdee58a26d	Add comprehensive database discovery outputs and enhance headless discovery - Add DATABASE_DISCOVERY_REPORT.md: Complete multi-agent database discovery findings covering structure, statistics, business domain, and query analysis - Add DATABASE_QUESTION_CAPABILITIES.md: Showcase of 14 question categories answerable via the discovery system with examples - Enhance headless_db_discovery.py: Improve JSON parsing and error handling - Enhance headless_db_discovery.sh: Add better argument handling and validation	1 month ago
Rene Cannao	b627f836f5	Refactor: Reorganize headless discovery scripts to dedicated directory Move headless database discovery scripts from scripts/ to scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/ for better organization. Also update README to: - Focus only on ProxySQL Query MCP (remove generic database examples) - Use relative paths (./) instead of absolute paths - Simplify configuration documentation Files moved: - scripts/HEADLESS_DISCOVERY_README.md - scripts/headless_db_discovery.py - scripts/headless_db_discovery.sh	1 month ago
Rene Cannao	d73ce0c41e	Add headless database discovery scripts Implement scripts for running Claude Code in non-interactive mode to perform comprehensive database discovery on any database. Files added: - headless_db_discovery.sh: Bash script implementation - headless_db_discovery.py: Python script implementation (recommended) - HEADLESS_DISCOVERY_README.md: Comprehensive documentation Features: - Works with any database accessible via MCP - Database-agnostic discovery prompt - Comprehensive analysis: structure, data, semantics, performance - Markdown report output with ERD, data quality score, recommendations - CI/CD integration ready - Supports custom MCP server configuration - Configurable timeout, output, verbosity Usage: python scripts/headless_db_discovery.py --database mydb	1 month ago
Rene Cannao	f852900365	Fix: Correct MCP catalog JSON parsing to handle special characters The catalog_search() and catalog_list() methods in MySQL_Catalog.cpp were manually building JSON strings by concatenating raw TEXT from SQLite without proper escaping. This caused parse errors when stored JSON contained quotes, backslashes, or newlines. Changes: - MySQL_Catalog.cpp: Use nlohmann::json to build proper nested JSON in search() and list() methods instead of manual concatenation - MySQL_Tool_Handler.cpp: Add try-catch for JSON parsing in catalog_get() - test_catalog.sh: Fix MCP URL path, add jq extraction for MCP protocol responses, add 3 special character tests (CAT013-CAT015) Test Results: All 15 catalog tests pass, including new tests that verify special characters (quotes, backslashes) are preserved.	1 month ago
Rene Cannao	1d046148d4	Fix: Address code review feedback from coderabbitai and gemini-code-assist Critical fixes: - Remove stray backslash at start of discover_cli.py (causes syntax error) - Fix escaped newlines (\\n) to real newlines (\n) in discover_cli.py Documentation fixes: - Update README.md: remove incorrect requirements_cli.txt reference - Update README.md: use generic path placeholder instead of /home/rene/... - Update STDIO_BRIDGE_README.md: mark PROXYSQL_MCP_ENDPOINT as optional with default Dependency updates: - Update package versions: httpx 0.27.0->0.28.1, pydantic 2.8.2->2.12.5, python-dotenv 1.0.1->1.2.1, rich 13.7.1->14.2.0 Per review: https://github.com/ProxySQL/proxysql-vec/pull/10	1 month ago
Rene Cannao	606fe2e93c	Fix: Address code review feedback from gemini-code-assist Python bridge (scripts/mcp/proxysql_mcp_stdio_bridge.py): - Make log file path configurable via PROXYSQL_MCP_BRIDGE_LOG env var - Add httpx.RequestError exception handling for network issues - Fix asyncio.CancelledError not being re-raised (HIGH priority) - Replace deprecated asyncio.get_event_loop() with get_running_loop() C++ server (lib/MCP_Endpoint.cpp): - Refactor handle_tools_call() to reduce code duplication - Handle string responses directly without calling .dump() - Single shared wrapping block for all response types Per review: https://github.com/ProxySQL/proxysql-vec/pull/11	1 month ago
Rene Cannao	2ceaac049c	docs: Add logging section to bridge README Added documentation for: - Log file location (/tmp/proxysql_mcp_bridge.log) - What information is logged - How to use logs for debugging	1 month ago
Rene Cannao	49e964bb02	Fix: Make ProxySQL MCP server return MCP-compliant tool responses The ProxySQL MCP server now wraps tool results in the correct MCP format: - result.content: array of content items (type: "text", text: "...") - result.isError: boolean Per MCP spec: https://modelcontextprotocol.io/specification/2025-11-25/server/tools Also simplified the bridge to pass through results directly since the server now returns the correct format.	1 month ago
Rene Cannao	9b4aea047a	Fix: Wrap tools/call responses in MCP-compliant content format Per MCP spec (https://modelcontextprotocol.io/specification/2025-11-25/server/tools): - Tool call responses MUST have a "content" array - Each content item has "type" and "text" fields - Response includes "isError" boolean This was the root cause - Claude Code was waiting for the correct format.	1 month ago
Rene Cannao	77099f7af2	Debug: Add minimal logging to track stdout writes and tool calls Added _log() calls to track: - stdout writes (bytes and content preview) - tools/call handler (name, response, result) - main startup Log is written to /tmp/proxysql_mcp_bridge.log	1 month ago
Rene Cannao	a47567fee7	Revert: Restore original bridge completely Restore to exact original code from commit `01c182cc`. The original code is clean and simple - just passes through responses. Only added minimal file-based logging for debugging.	1 month ago
Rene Cannao	23e5efca5a	Test: Don't redirect sys.stderr, write logs directly to file The sys.stderr redirection may be interfering with stdout/stdin pipes. Now logging writes directly to a file handle instead of through sys.stderr.	1 month ago
Rene Cannao	f4a4af8d82	Fix: Write directly to stdout.buffer to bypass TextIOWrapper issues The TextIOWrapper may have buffering issues when stderr is redirected. Writing directly to the binary buffer with encoded bytes ensures immediate delivery of responses to Claude Code.	1 month ago
Rene Cannao	ad54f92dc5	Revert: Simplify tool handlers back to original pass-through Remove all the unwrapping and TextContent wrapping logic that was added. Go back to the original simple pass-through that just returns the result from ProxySQL directly. The original format was correct.	1 month ago
Rene Cannao	2b5134632c	Fix: Wrap tool results in TextContent format for MCP protocol compliance The MCP protocol requires tool call results to be wrapped in content items with type and text fields. This matches what other MCP servers do. Before: {"result": [{"name": "testdb", ...}]} After: {"result": [{"type": "text", "text": "[{\"name\": \"testdb\", ...}]"}]} This should fix the issue where Claude Code was timing out waiting for responses.	1 month ago
Rene Cannao	55dd5ba574	Debug: Add detailed stdout write logging to troubleshoot Claude Code timeout - Revert the stdout replacement changes (was probably not the issue) - Add detailed logging to _writeline to see exactly what's happening when writing to stdout	1 month ago
Rene Cannao	f5606986ff	Fix: Replace stdout with truly unbuffered wrapper to prevent response buffering The previous sys.stdout.reconfigure(line_buffering=True) didn't work when stderr is redirected. Now we create a new io.TextIOWrapper around sys.stdout.buffer with line_buffering=False, ensuring immediate flush. Also sets PYTHONUNBUFFERED=1 for extra safety.	1 month ago
Rene Cannao	edac8eb5e0	Fix: Add verbose logging and fix stdout buffering issue in MCP stdio bridge - Redirect stderr to /tmp/proxysql_mcp_bridge.log for debugging - Add extreme verbosity with timestamps for all stdin/stdout/HTTP traffic - CRITICAL FIX: Set stdout to line-buffered mode to prevent responses from being buffered and never reaching Claude Code (causing timeouts) - Log all HTTP requests/responses to ProxySQL MCP server - Log all message handling and unwrapping operations	1 month ago
Rene Cannao	fc6b462be1	Fix: unwrap ProxySQL nested response format ProxySQL MCP wraps tool responses in {"result": {...}, "success": true}. The bridge now unwraps this to return just the actual result to Claude Code. This fixes the LLM error 'The prompt parameter was not received normally' which occurred because the LLM was receiving the malformed nested structure.	1 month ago
Rene Cannao	4491f3ce0b	Add debug logging to MCP bridge for troubleshooting Add PROXYSQL_MCP_DEBUG environment variable to enable verbose logging of all stdio communication and ProxySQL HTTP requests/responses.	1 month ago
Rene Cannao	01c182ccac	Add stdio MCP bridge for Claude Code integration Add a Python stdio-based MCP server that bridges to ProxySQL's HTTPS MCP endpoint, enabling Claude Code to use ProxySQL MCP tools directly. The bridge: - Implements stdio MCP server protocol (for Claude Code) - Acts as MCP client to ProxySQL's HTTPS endpoint - Supports initialize, tools/list, tools/call methods - Handles authentication via Bearer tokens - Configurable via environment variables Usage: - Configure in Claude Code MCP settings - Set PROXYSQL_MCP_ENDPOINT environment variable - Optional: PROXYSQL_MCP_TOKEN for auth	1 month ago
Rene Cannao	9d6a2173bf	Enhance Rich CLI with configurable LLM chat path and better tracing LLM improvements: - Add configurable chat path (LLM_CHAT_PATH or --llm-chat-path) to support non-standard endpoints like Z.ai's /api/coding/paas/v4 - Add optional JSON mode (LLM_JSON_MODE or --llm-json-mode) for models that support native JSON output - Enhanced tracing: log HTTP status and response body snippet on every request - Safer JSON parsing: treat empty content as error with helpful message - Better error messages with diagnostic hints Code cleanup: - Remove intent command (simplify CLI) - Remove user_intent reading and passing - Simplify stopping logic (just run max_iterations) - Clean up formatting and remove unused code	1 month ago
Rene Cannao	f2ca750c05	Add MCP Database Discovery Agent (initial commit) Add a database discovery agent prototype that uses LLMs to explore databases through the MCP Query endpoint. Includes: - Rich CLI (discover_cli.py): Working async CLI with Rich TUI, proper MCP tools/call JSON-RPC method, and full tracing support - FastAPI_deprecated_POC: Early prototype with incorrect MCP protocol, kept for reference only The Rich CLI version implements a multi-expert agent architecture: - Planner: Chooses next tasks based on state - Structural Expert: Analyzes table structure and relationships - Statistical Expert: Profiles tables and columns - Semantic Expert: Infers domain meaning - Query Expert: Validates access patterns	1 month ago
Rene Cannao	ef5b99edbf	Fix MCP tool bugs: NULL value handling and query validation - Fixed NULL value handling in execute_query: use empty string instead of nullptr to avoid "basic_string: construction from null" errors - Fixed validate_readonly_query: corrected substring length check from substr(0,6)!="SELECT " to substr(0,6)!="SELECT" - Fixed test script: added proper variable_name parameter for get_config/set_config tools Query endpoint tools now pass all tests.	1 month ago
Rene Cannao	de33d177be	Fix verbose mode in test_mcp_tools.sh Verbose output was being printed to stdout, contaminating the captured response value and causing tool discovery to fail. Redirect debug output to stderr instead.	1 month ago
Rene Cannao	25cda31f08	Update test_mcp_tools.sh with dynamic tool discovery - Add discover_tools() function that calls tools/list on each endpoint - Store discovered tools in temp file to avoid bash associative array issues - Define all expected tools with test configurations - Only test tools that are discovered via tools/list - Add support for all 5 endpoints: config, query, admin, cache, observe - Add --list-only flag to show discovered tools without testing - Add --endpoint flag to test specific endpoint - Improve help output with endpoint descriptions	1 month ago
Rene Cannao	093511920f	Add environment variable printing to MCP scripts Print environment variables at the start of each script when they are set: - setup_test_db.sh: Shows MYSQL_HOST, MYSQL_PORT, MYSQL_USER, MYSQL_PASSWORD, TEST_DB_NAME - configure_mcp.sh: Shows MYSQL_HOST, MYSQL_PORT, MYSQL_USER, MYSQL_PASSWORD, TEST_DB_NAME, MCP_PORT - test_mcp_tools.sh: Shows MCP_HOST, MCP_PORT This helps users verify that the correct environment variables are being used.	1 month ago
Rene Cannao	7f957088ee	Fix configure_mcp.sh to allow empty MySQL passwords Change MYSQL_PASSWORD from using :- to = for default value. This allows setting an empty password via environment variable: - MYSQL_PASSWORD="" will now use empty password - Previously, empty string was treated as unset, forcing the default	1 month ago
Rene Cannao	49e6ac5bc6	Revert configure_mcp.sh to respect environment variables The script correctly uses ${VAR:-default} syntax, so it already respects environment variables. The issue was with the MCP server not reinitializing when MySQL configuration changes, which is a separate issue.	1 month ago
Rene Cannao	40cff23c3b	Initialize MySQL Tool Handler and fix default MySQL port - Initialize MySQL_Tool_Handler in ProxySQL_MCP_Server constructor with MySQL configuration from MCP variables - Use GloVars.get_SSL_pem_mem() to get SSL certificates correctly - Add MySQL_Tool_Handler cleanup in destructor - Change configure_mcp.sh default MySQL port from 3307 to 3306 - Change configure_mcp.sh default password from test123 to empty - Update help text and examples to match new defaults	1 month ago
Rene Cannao	aeafa61a11	Fix test_mcp_tools.sh to use correct MCP endpoint paths - Change /config to /mcp/config - Change /query to /mcp/query	1 month ago
Rene Cannao	d17fe1dba8	Fix configure_mcp.sh error handling and endpoint paths - Split exec_admin into exec_admin (shows errors) and exec_admin_silent - Execute each SET command individually to catch specific failures - Add proper error checking and early exit on configuration failures - Fix MCP endpoint test URL from /config to /mcp/config - Update displayed endpoint URLs to include /mcp/ prefix and all 5 endpoints	1 month ago
Rene Cannao	60d4a7378c	Implement automatic MCP server start/stop and add environment variable support - Add automatic MCP HTTPS server start/stop based on mcp-enabled flag - Server starts when mcp-enabled=true and LOAD MCP VARIABLES TO RUNTIME - Server stops when mcp-enabled=false and LOAD MCP VARIABLES TO RUNTIME - Validates SSL certificates before starting - Added to both flush_mcp_variables___database_to_runtime() and flush_mcp_variables___runtime_to_database() functions - Update configure_mcp.sh to respect environment variables - MYSQL_HOST, MYSQL_PORT, MYSQL_USER, MYSQL_PASSWORD - TEST_DB_NAME (mapped to MYSQL_DATABASE) - MCP_PORT - Updated --help documentation with all supported variables	1 month ago
Rene Cannao	28742554b5	Use relative catalog path instead of absolute path - Change mcp-catalog_path default from /var/lib/proxysql/mcp_catalog.db to mcp_catalog.db - SQLite accepts relative paths, which are resolved relative to the process working directory - ProxySQL's working directory is its datadir, so the catalog will be stored there - Update configure_mcp.sh to set mcp-catalog_path='mcp_catalog.db' - Update lib/MCP_Thread.cpp default to "mcp_catalog.db" - Update README.md to document relative path behavior	1 month ago
Rene Cannao	c53b28e42a	Add comprehensive documentation to MCP README - Add architecture overview with ASCII diagram showing all components - Add detailed component explanations (ProxySQL MCP Module, Connection Pool, Catalog, Test Scripts) - Add testing flow diagram with 4-step process - Add Quick Start section with copy/paste commands for native and Docker modes - Add detailed documentation for each script explaining what they do and why - Add troubleshooting section for common issues - Add default configuration reference table - Add environment variables reference This allows users to understand the architecture and run tests without needing to understand implementation details.	1 month ago
Rene Cannao	b3646b4798	Fix argument parsing and documentation in setup_test_db.sh Fixed critical issues with argument parsing that prevented the script from working correctly: 1. Fixed argument order - script now supports both: - ./setup_test_db.sh <command> [options] - ./setup_test_db.sh [options] <command> 2. Fixed --help option - now shows help instead of running commands 3. Updated README.md examples with correct syntax: - OLD: ./setup_test_db.sh --mode native start (wrong) - NEW: ./setup_test_db.sh start --mode native (correct) The script now properly: - Parses -h/--help anywhere and shows usage - Handles options before or after the command - Auto-detects mode when not specified - Shows helpful connection info after setup All examples in README.md updated with correct command syntax.	1 month ago
Rene Cannao	3d827144e8	Add required environment variables section to README Added comprehensive environment variable documentation to the README Prerequisites section to make it clear what needs to be configured before running the test scripts. Changes: - Added "Required Environment Variables" subsection under Prerequisites - Listed all ProxySQL Admin, MySQL, and MCP server environment variables - Added quick setup example for adding variables to ~/.bashrc - Made it clear that PROXYSQL_ADMIN_PASSWORD and MYSQL_PASSWORD need to be set Users now have clear guidance on required configuration before testing.	1 month ago
Rene Cannao	ad2e2a24d1	Add native MySQL mode support to test database setup Updated setup_test_db.sh to support both Docker and native MySQL modes. Changes: - Added --mode option: docker, native, or auto (default: auto-detect) - Auto-detect: tries Docker first, then falls back to native MySQL - Native mode functions: start_native, status_native, connect_native, reset_native - Native mode connects to existing MySQL server (no Docker required) - Added --host, --port, --user, --password, --database options - Environment variable support: MYSQL_HOST, MYSQL_PORT, MYSQL_USER, MYSQL_PASSWORD Updated README.md with: - Native mode quick start guide - Docker mode quick start guide - Auto-detect mode documentation - Comprehensive setup_test_db.sh usage examples - Environment variable documentation Usage examples: # Auto-detect mode ./setup_test_db.sh start # Native MySQL mode ./setup_test_db.sh --mode native --host localhost --port 3306 start # Docker mode ./setup_test_db.sh --mode docker start	1 month ago
Rene Cannao	e9a6dd0b3e	Add comprehensive MCP testing suite in scripts/mcp/ Created a complete testing suite for the MCP module with MySQL connection pool and exploration tools. Files added: - README.md: Comprehensive testing documentation - setup_test_db.sh: Docker-based test MySQL database setup - Start/stop/status/connect commands - Creates sample schema (customers, orders, products, order_items) - Includes views and stored procedures for testing - configure_mcp.sh: ProxySQL MCP module configuration - Configures MySQL connection parameters - Enables/disables MCP server - Shows current configuration status - test_mcp_tools.sh: Main MCP tools test suite - Tests all 15 MCP tools (list_schemas, list_tables, etc.) - Includes catalog tests (upsert, get, search, delete) - Reports pass/fail statistics - stress_test.sh: Concurrent connection stress testing - Configurable number of concurrent requests - Response time measurement - Success rate calculation - test_catalog.sh: Catalog/LLM memory specific tests - 12 catalog operation tests - FTS search testing - CRUD verification All scripts are executable and include: - Command-line argument parsing - Colored output for readability - Error handling and validation - Usage/help documentation - Environment variable support	1 month ago
Rene Cannao	62cbd6c71e	Fix issues identified in AI code review - Fix combined_search: Build SQL query dynamically based on search_term - Only include MATCH relevance when search_term is provided - Fix parameter binding (fulltext_params vs params) - Fix search_term/query parameter mismatch in run_demo - combined_search mode now correctly receives query from kwargs - Add environment variable support for database credentials - nlp_search_demo.py: DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME - stackexchange_posts.py: SOURCE_DB_* and TARGET_DB_* variables - Fix README CreationDate SQL example to use JSON_EXTRACT - Add zero division checks in get_table_stats and similarity_search_preparation	2 months ago
Rene Cannao	ecfff09633	Add NLP search demo script with comprehensive search capabilities - Create nlp_search_demo.py with full-text, boolean, tag-based, and combined search - Include statistics and similarity search preparation modes - Add proper error handling with MySQL fallback mechanisms - Fix syntax errors and indentation issues - Update README.md with correct command usage examples - Support --mode parameter with appropriate sub-parameters - Demonstrate various search techniques on processed StackExchange posts	2 months ago
Rene Cannao	d37d291488	Implement comprehensive StackExchange posts processing with search capabilities - Unified script that creates target table with search-optimized schema - Adds full-text search indexes and text processing for better search results - Implements duplicate detection and prevention - Fixed MySQL functional index compatibility issue (removed problematic Tags index) - Added support for --limit 0 to process all posts - Added --warning-large-batches flag for batch size > 1000 - Improved text cleaning and normalization for search optimization - Enhanced progress tracking and batch processing efficiency Removes separate scripts (populate_search_columns.py, nlp_search_demo.py) and consolidates all functionality into unified solution.	2 months ago
Rene Cannao	d94dc036e8	Add StackExchange posts processing script with JSON storage - Retrieves parent posts (PostTypeId=1) and their replies (PostTypeId=2) from MySQL - Combines posts and tags into structured JSON format - Supports batch processing with memory-efficient chunking - Includes duplicate checking to avoid reprocessing - Creates target table automatically with JSON storage and BLOB for embeddings - Handles large datasets with offset-based pagination - Optimized to skip duplicate processing work - Includes verification and progress tracking Files: - scripts/stackexchange_posts.py: Main processing script	2 months ago
Rene Cannao	221831afc1	Add usage examples to script documentation and help output Changes: 1. Add comprehensive usage examples to script docstring with: - Remote API example (requires API_KEY environment variable) - Local Ollama example (uses --local-ollama flag) - Both examples show all common command line options 2. Add epilog to argparse help with concise examples: - Shows minimal command line for both modes - Points to docstring for full examples - Uses RawDescriptionHelpFormatter for proper formatting Users now have multiple ways to access usage information: - Read script header with `head -50 scripts/process_posts_embeddings.py` - Run `python3 scripts/process_posts_embeddings.py --help` - Both show appropriate examples for remote API and local Ollama modes	2 months ago
Rene Cannao	4aba7137b4	Add --local-ollama option for local Ollama server support New option: --local-ollama - Uses Ollama format with localhost:11434 API endpoint - Model: nomic-embed-text-v1.5 (without hf: prefix) - No API_KEY environment variable required - Overrides api-format, api-url, and api-model flags Changes: 1. Add --local-ollama boolean flag to parse_args() 2. Modify check_env() to skip API_KEY check when local-ollama is set 3. Update configure_client() to generate Ollama-specific SQL without 'key' parameter 4. Update main() to display correct configuration based on mode 5. Update documentation with local Ollama usage Behavior: - Without --local-ollama: Requires API_KEY, uses remote API with configurable format/url/model - With --local-ollama: No API_KEY needed, uses fixed local Ollama configuration	2 months ago

1 2

58 Commits (3fe8a48f705bea9bb77c3255e54bbc1ace125ebb)