2 Commits (ffdb334dc30929103603f1bbdb02cdeed8f73373)

Author	SHA1	Message	Date
Rene Cannao	ffdb334dc3	Add WHERE filters to prevent empty input errors and fix SQL syntax ... Changes: - Filter Posts by PostTypeId IN (1,2) (Questions and Answers) - Filter by minimum text length > 30 characters (Title + Body) - Update get_total_posts to count only eligible posts for accurate progress - Fix SQL syntax error in process_batch WHERE clause - Update documentation with filter details Rationale: - Empty or very short text causes embedding generation failures - PostTypeId 1,2 are most relevant content (Questions and Answers) - Ensures consistent counting between total, remaining, and processed	2 months ago
Rene Cannao	36a59f3f56	Add Posts embeddings processing script with exponential backoff ... New script: scripts/process_posts_embeddings.py Features: - Connects to SQLite3 server via MySQL connector with configurable credentials - Configures API client using API_KEY environment variable (fails if not set) - Processes unembedded Posts rows in configurable batch sizes (default: 10) - Uses LEFT JOIN WHERE IS NULL pattern to track unprocessed rows - Implements exponential backoff for retry failures with 5-minute maximum cap - Shows progress: remaining rows, processed count, percentage complete - Fails if Posts_embeddings table doesn't exist (no automatic creation) - Handles concurrent processing race conditions with small delays Script prerequisites: 1. Posts table must exist (copied from MySQL) 2. Posts_embeddings virtual table must exist: CREATE VIRTUAL TABLE Posts_embeddings USING vec0(embedding float[768]); Backoff behavior: - Default retry delay: 5 seconds - Exponential increase: 5s, 10s, 20s, 40s, ... up to 300s maximum - Resets on any successful operation (even if 0 rows processed)	2 months ago