4 Commits (fb5470c4117bbc3136ee5656edfd8a7463a2192a)

Author	SHA1	Message	Date
Rene Cannao	221831afc1	Add usage examples to script documentation and help output ... Changes: 1. Add comprehensive usage examples to script docstring with: - Remote API example (requires API_KEY environment variable) - Local Ollama example (uses --local-ollama flag) - Both examples show all common command line options 2. Add epilog to argparse help with concise examples: - Shows minimal command line for both modes - Points to docstring for full examples - Uses RawDescriptionHelpFormatter for proper formatting Users now have multiple ways to access usage information: - Read script header with `head -50 scripts/process_posts_embeddings.py` - Run `python3 scripts/process_posts_embeddings.py --help` - Both show appropriate examples for remote API and local Ollama modes	5 months ago
Rene Cannao	4aba7137b4	Add --local-ollama option for local Ollama server support ... New option: --local-ollama - Uses Ollama format with localhost:11434 API endpoint - Model: nomic-embed-text-v1.5 (without hf: prefix) - No API_KEY environment variable required - Overrides api-format, api-url, and api-model flags Changes: 1. Add --local-ollama boolean flag to parse_args() 2. Modify check_env() to skip API_KEY check when local-ollama is set 3. Update configure_client() to generate Ollama-specific SQL without 'key' parameter 4. Update main() to display correct configuration based on mode 5. Update documentation with local Ollama usage Behavior: - Without --local-ollama: Requires API_KEY, uses remote API with configurable format/url/model - With --local-ollama: No API_KEY needed, uses fixed local Ollama configuration	5 months ago
Rene Cannao	ffdb334dc3	Add WHERE filters to prevent empty input errors and fix SQL syntax ... Changes: - Filter Posts by PostTypeId IN (1,2) (Questions and Answers) - Filter by minimum text length > 30 characters (Title + Body) - Update get_total_posts to count only eligible posts for accurate progress - Fix SQL syntax error in process_batch WHERE clause - Update documentation with filter details Rationale: - Empty or very short text causes embedding generation failures - PostTypeId 1,2 are most relevant content (Questions and Answers) - Ensures consistent counting between total, remaining, and processed	5 months ago
Rene Cannao	36a59f3f56	Add Posts embeddings processing script with exponential backoff ... New script: scripts/process_posts_embeddings.py Features: - Connects to SQLite3 server via MySQL connector with configurable credentials - Configures API client using API_KEY environment variable (fails if not set) - Processes unembedded Posts rows in configurable batch sizes (default: 10) - Uses LEFT JOIN WHERE IS NULL pattern to track unprocessed rows - Implements exponential backoff for retry failures with 5-minute maximum cap - Shows progress: remaining rows, processed count, percentage complete - Fails if Posts_embeddings table doesn't exist (no automatic creation) - Handles concurrent processing race conditions with small delays Script prerequisites: 1. Posts table must exist (copied from MySQL) 2. Posts_embeddings virtual table must exist: CREATE VIRTUAL TABLE Posts_embeddings USING vec0(embedding float[768]); Backoff behavior: - Default retry delay: 5 seconds - Exponential increase: 5s, 10s, 20s, 40s, ... up to 300s maximum - Resets on any successful operation (even if 0 rows processed)	5 months ago