13 Commits (048e4e85d5ceb6bbb9f6a6d2b2efbc259cc331ff)

Author	SHA1	Message	Date
Broque Thomas	115d7ed9c5	Preserve personalized playlist metadata for wishlist	2 weeks ago
Broque Thomas	d861a40277	Personalized pipeline: refresh snapshot on first-run too ... Reproduced: selecting Fresh Tape (or any kind never generated before) and running the pipeline silently skipped — UI showed "No tracks in Fresh Tape — skipping sync" with no clue why. Root cause: ensure_playlist auto-creates the playlist row on first access with `track_count=0` and `last_generated_at=NULL`, but `is_stale=0` by default (the column default — fresh rows aren't "stale", they're "never generated"). Pipeline only refreshed when `is_stale=True` OR `refresh_first=True`, so first-run rows fell through both branches → read the empty snapshot → skip. Fix: pipeline now also refreshes when `existing.last_generated_at is None`. Same control flow, one extra condition: if refresh_first OR is_stale OR last_generated_at is None: refresh else: read existing snapshot This is the right signal: "has the generator ever run for this row" is exactly what `last_generated_at` tracks (the column is set in `_persist_snapshot` after every successful refresh). Stubs in test_handlers_personalized_pipeline.py updated to expose `last_generated_at` on their SimpleNamespace returns so the new attribute read doesn't AttributeError. Fresh stubs get a non-None timestamp so they're treated as already-generated; the new test `test_never_generated_snapshot_triggers_first_refresh` pins the first-run-forces-refresh behavior with `last_generated_at=None`.	2 weeks ago
Broque Thomas	877d0e7d81	Personalized pipeline: auto-refresh stale snapshots after watchlist scan ... Snapshots now track when their source data changes. Watchlist scan emits stale flags on the playlists whose underlying pool just got refreshed; the next pipeline run sees the flag and regenerates the snapshot before syncing, so the server playlist never lags the source. Schema: - new `is_stale INTEGER NOT NULL DEFAULT 0` column on `personalized_playlists`, plus an idempotent ADD COLUMN migration in `ensure_personalized_schema` for installs created before this PR. - `PlaylistRecord.is_stale: bool = False` exposed on the dataclass so callers can branch on freshness without re-querying. Manager: - new `mark_kinds_stale(kinds, profile_id=None)` flips the flag in bulk for a list of kinds (used by upstream data refreshers). - `_persist_snapshot` clears `is_stale = 0` on successful refresh. - SELECT statements + `_row_to_record` updated to read the column (with tuple-form length guard for safety). Pipeline: - `_build_payloads_for_kinds` now branches: refresh_first=True OR `existing.is_stale` -> refresh_playlist, else read existing snapshot. So the auto-refresh kicks in without needing the user to toggle the refresh-each-run option. Watchlist scanner emits stale flags at three sites: - after `update_discovery_pool_timestamp` -> marks pool-fed kinds stale: hidden_gems, discovery_shuffle, popular_picks, time_machine, genre_playlist, daily_mix. - after release_radar `save_curated_playlist` -> marks `fresh_tape`. - after discovery_weekly `save_curated_playlist` -> marks `archives`. All three calls go through a module-level `_mark_personalized_kinds_stale` helper that builds a PersonalizedPlaylistManager with `deps=None` (only DB access is needed for the flag update — no generator dispatch). Each call is wrapped in try/except so a flag failure can never abort the scan itself. Tests: - new `TestStaleFlag` class in `test_personalized_manager.py` (6 tests): default-false, single-kind flip, multi-kind, profile scoping, refresh-clears, empty-list noop. - two new pipeline tests pin the auto-refresh dispatch: `test_stale_snapshot_auto_refreshes_even_without_refresh_first` and `test_non_stale_snapshot_skips_refresh`. - existing stub-manager `SimpleNamespace` returns gained `is_stale=False` so the new attribute read doesn't AttributeError. Full suite: 3391 pass. User-facing WHATS_NEW entry added under 2.5.2 (above the prior pipeline auto-sync entry) describing the auto-refresh behavior.	2 weeks ago
Broque Thomas	e1f0810df5	Personalized pipeline: UI multi-select picker for kinds + variants ... The action was registered + the block declared, but the automation builder's per-action config renderer didn't have a case for `personalized_pipeline` so users only saw the bare card with the generic delay-minutes input — no way to select which playlists to sync. This commit adds the multi-select picker. Backend: - `core/personalized/api.list_kinds(manager=...)` now optionally takes a manager and includes the resolved variant list per kind (calls each spec's variant_resolver(deps) when present). Singleton kinds get an empty `variants` list. Variant-bearing kinds (time_machine / genre_playlist / daily_mix / seasonal_mix) get their full enumerated set. - `web_server.py` `/api/personalized/kinds` route now passes a built manager so the variants list lands in the response. Frontend: - `webui/static/stats-automations.js` `_renderBlockConfigFields` gains a `personalized_pipeline` branch that renders a scrollable multi-select picker: - Singletons (Hidden Gems, Discovery Shuffle, Popular Picks, Fresh Tape, The Archives) = one checkbox row per kind - Variant kinds = a section header + one checkbox row per variant (e.g. Time Machine: 1960s/1970s/.../2020s; Seasonal: halloween/ christmas/valentines/summer/spring/autumn) - Pre-checks rows that match the existing `kinds` config on edit - New `_autoLoadPersonalizedKinds(slotKey)` fetches `/api/personalized/kinds` (cached after first load), renders the picker DOM, and pre-checks saved selections via `data-kind` / `data-variant` attributes on the checkboxes. - `_renderBuilderCanvas` calls the loader for any `cfg-*-kinds-picker` it finds in the freshly-rendered slots. - The save-time `_collectActionConfig` walks the picker's checked inputs (matched by `data-kind` attribute) and emits `{kinds: [{kind, variant?}, ...], refresh_first, skip_wishlist}` in the same shape the handler expects. Tests: - `tests/automation/test_automation_blocks.py::_FIELD_TYPES` adds 'personalized_playlist_select' so the block-shape regression test accepts the new field type. (Test was failing because it whitelists every field type used across all blocks.) - 189 automation + personalized API tests pass; full suite intact.	2 weeks ago
Broque Thomas	cc44254bf9	Personalized playlist pipeline: auto-sync discover-page playlists ... Follow-up to the personalized-playlists standardization PR. New `personalized_pipeline` automation action syncs selected discover- page playlists (Hidden Gems / Discovery Shuffle / Time Machine / Genre / Daily Mix / Fresh Tape / The Archives / Seasonal Mix) to the active media server + queues missing tracks for download. Same pattern as the existing mirrored `playlist_pipeline` but two phases instead of four — no REFRESH (no external source to re-pull) and no DISCOVER (manager-backed snapshots are already metadata- matched). Pipeline shape: SNAPSHOT → SYNC → WISHLIST Where SNAPSHOT either reads the persisted track list from `PersonalizedPlaylistManager` (default) or refreshes it first when `refresh_first=true` (cron use case: regenerate Hidden Gems nightly and sync the fresh set). Shared helper extraction: PHASE 3 (SYNC loop) + PHASE 4 (WISHLIST tail) lifted out of mirrored `playlist_pipeline` into `core/automation/handlers/_pipeline_shared.py` as `run_sync_and_wishlist(deps, automation_id, playlists, sync_one_fn, sync_id_for_fn, ...)`. Both pipelines call it. Mirrored injects `auto_sync_playlist` as the per-playlist sync function; personalized injects a thin wrapper that launches `_run_sync_task` directly with a pre-built tracks_json. Same sync-state polling / progress emission / status counting / wishlist trigger logic — 0 duplication. Files added: - core/automation/handlers/_pipeline_shared.py - core/automation/handlers/personalized_pipeline.py - tests/automation/test_handlers_personalized_pipeline.py Files changed: - core/automation/handlers/playlist_pipeline.py: PHASE 3+4 replaced with shared helper call (~100 lines deleted, 1 helper invocation added; behavior identical). - core/automation/deps.py: new `build_personalized_manager` field (lazy builder so the pipeline gets a fresh PersonalizedPlaylistManager per run). - core/automation/handlers/__init__.py + registration.py: register `personalized_pipeline` action with the shared `pipeline_running` guard so it can't overlap mirrored. - core/automation/blocks.py: new `personalized_pipeline` block declaration with config_fields (kinds multi-select, refresh_first, skip_wishlist). - web_server.py: thread `_build_personalized_manager` into AutomationDeps construction. - All 5 automation test fixtures: `_build_deps` adds `build_personalized_manager=lambda: None` stub. - tests/automation/test_handler_registration.py: EXPECTED_ACTION_NAMES + EXPECTED_GUARDED_ACTIONS gain `personalized_pipeline`. Trigger schema: { "_automation_id": "...", "kinds": [ {"kind": "hidden_gems"}, {"kind": "time_machine", "variant": "1980s"}, {"kind": "seasonal_mix", "variant": "halloween"} ], "refresh_first": false, "skip_wishlist": false } Tests (14 new, 178 automation total): - _track_to_sync_shape: basic shape, source ID fallback chain, no-id returns empty string - empty config / non-list kinds / empty kinds list all return error + clear pipeline_running flag - _build_payloads_for_kinds: skips invalid entries, skips kinds with no tracks, refresh_first vs ensure dispatch, payload shape + sync_id format, manager exception swallowed continues - _sync_personalized_playlist: launches background thread + returns status='started' - happy path: stubbed sync_states drives helper to completion, flag cleaned up Full suite: 3383 passed. Note: the trigger UI block declares config_fields but the frontend doesn't yet render the `personalized_playlist_select` multi-select type — usable today via API; polished UI ships in a follow-up frontend PR.	2 weeks ago
Broque Thomas	d3768610d7	Extract automation handlers (kettui-bar): engine-boundary tests ... Per-handler boundary tests pin each handler's body in isolation. Adding engine-boundary tests that pin the REGISTRATION layer: - every expected action name registered, no drops, no extras - guarded actions register a guard, unguarded ones don't - every registered handler is callable - every guard returns a bool - all four progress callbacks registered in the right slots - progress_init / progress_finish / record_history / on_library_scan_completed are invocable through the engine's stored callable shape (not just the bare extracted function) - finish callback respects _manages_own_progress flag at the engine boundary too - library_scan_completed wiring registers a callback on the scan manager and that callback fires engine.emit when invoked - every handler returns a `{'status': ...}` dict on a minimal config trigger -- proves no handler raises into the engine, even when its guard / short-circuit / error path is the one taken Uses a minimal _RecordingEngine that captures registrations + a _RecordingScanMgr that captures completion callbacks. No real AutomationEngine, no real Flask app, no real DB. The kettui standard for refactor PRs: don't ship "behavior preserved" claim that's only validated at the function boundary -- exercise the engine seam too. EXPECTED_ACTION_NAMES + EXPECTED_GUARDED_ACTIONS frozen sets at the top: any future drift (rename / drop / add a handler / change which ones are guarded) fails this test immediately so refactor PRs can't quietly mutate the registration shape. 13 new tests, 164 automation tests pass total.	2 weeks ago
Broque Thomas	e140da117a	Extract automation handlers (4/3 — finish): progress callbacks + scan-completion emitter ... Cleans up the four remaining inline callbacks at the bottom of `web_server._register_automation_handlers` so the function is now purely deps-construction + register_all + a logger.info line. Lifted: - `_progress_init`, `_progress_finish`, `_record_automation_history`, and `_on_library_scan_completed` -> core/automation/handlers/progress_callbacks.py Each is a top-level function that takes deps as a parameter; the engine sees thin lambdas through `register_progress_callbacks` / `register_library_scan_completed_emitter` (called from `register_all`). Two new deps fields: - `init_automation_progress` (delegates into the live progress tracker) - `record_progress_history` (delegates into _auto_progress.record_history) 12 new boundary tests in tests/automation/test_progress_callbacks.py pin every shape: - progress_init forwards to init_automation_progress - progress_finish skips when handler manages its own progress (prevents double-emit of finished status) - progress_finish: completed -> finished/Complete/success; error -> error/Error/error; msg falls through error -> reason -> status -> 'done' - record_history threads the live db into the recorder - on_library_scan_completed: no engine = noop, server type taken from web_scan_manager._current_server_type, defaults to 'unknown' - register_library_scan_completed_emitter: no scan manager = noop, registered callback emits the right event when invoked 3256 tests pass, no regression. Final state of `_register_automation_handlers`: - Was: 1530 lines, 21 nested closures + 4 progress callbacks - Now: ~50 lines, builds AutomationDeps and calls register_all web_server.py: 34,220 -> 34,187 lines (-33 net, -1,406 across the whole branch).	2 weeks ago
Broque Thomas	017553193f	Extract automation handlers (3/3): maintenance + misc, finishing the lift ... Final commit of the automation-handler refactor. With this commit every closure that used to live in `web_server._register_automation_handlers` is now a top-level function in `core/automation/handlers/`. Handlers extracted in this commit: - start_database_update + deep_scan_library -> core/automation/handlers/database_update.py Both share the db_update_state monitoring pattern (poll until status flips, stall detection emits warning at 10 min, 2-hour outer timeout). Lifted into a shared `_run_with_progress` helper inside the module so the per-handler bodies stay tiny. - run_duplicate_cleaner -> core/automation/handlers/duplicate_cleaner.py - start_quality_scan -> core/automation/handlers/quality_scanner.py - clear_quarantine, cleanup_wishlist, update_discovery_pool, backup_database, refresh_beatport_cache -> core/automation/handlers/maintenance.py Grouped because each body is short (~20-50 lines) and they share no state — splitting into per-handler files would just add import noise. - clean_search_history, clean_completed_downloads, full_cleanup -> core/automation/handlers/download_cleanup.py Grouped because all three reach the download orchestrator, tasks_lock, and download_batches/download_tasks accessors. The full_cleanup multi-step orchestration shares phase-detection logic with clean_completed_downloads. - run_script -> core/automation/handlers/run_script.py - search_and_download -> core/automation/handlers/search_and_download.py `AutomationDeps` grew with the new dependency surface: - get_db_update_state + db_update_lock + db_update_executor + run_db_update_task + run_deep_scan_task - get_duplicate_cleaner_state + duplicate_cleaner_lock + duplicate_cleaner_executor + run_duplicate_cleaner - get_quality_scanner_state + quality_scanner_lock + quality_scanner_executor + run_quality_scanner - download_orchestrator + run_async + tasks_lock + get_download_batches + get_download_tasks + sweep_empty_download_directories + get_staging_path - docker_resolve_path + get_current_profile_id + get_watchlist_scanner + get_app + get_beatport_data_cache - set_db_update_automation_id (writes the legacy global so the live DB-update progress callbacks still living in web_server.py keep emitting against the active automation card) `web_server._register_automation_handlers` is now ~50 lines: build deps once, call register_all. The 667-line block of remaining closure definitions and engine register calls is gone. The final orphan was the `_db_update_automation_id` module global — the DB-update progress callbacks at line ~14080 still read it directly, so the extracted database_update handler propagates the automation id through `deps.set_db_update_automation_id` (a closure in web_server that writes the global). When the legacy callbacks get extracted in a future PR the setter goes away. Tests: - tests/automation/test_handlers_maintenance.py adds 21 boundary tests covering every newly-extracted handler shape: guard short-circuits (already-running returns skipped), deps wiring (set_db_update_automation_id called with the right id), exception swallow contract, status returns, path-traversal blocked in run_script, source-mode skip in clean_search_history, active-batch skip in clean_completed_downloads, etc. - 3244 tests pass (was 3223 — 21 new), no regression. web_server.py: 35,593 -> 34,220 lines (-1,373 net across 3 commits). Issue #1 from the extraction punch list is now COMPLETE.	2 weeks ago
Broque Thomas	cde237c7e7	Extract automation handlers (2/N): playlist lifecycle group ... Continues the lift from `web_server._register_automation_handlers`. This commit extracts the four playlist-lifecycle closures: - `refresh_mirrored` -> core/automation/handlers/refresh_mirrored.py - `sync_playlist` -> core/automation/handlers/sync_playlist.py - `discover_playlist` -> core/automation/handlers/discover_playlist.py - `playlist_pipeline` -> core/automation/handlers/playlist_pipeline.py The pipeline composes refresh + sync + discover, so all four ship together. The pipeline imports the other three handler modules directly (cross-handler call) instead of going through the engine, preserving the "single trigger from the user's perspective" UX. `AutomationDeps` grew to cover the new dependency surface: - run_playlist_discovery_worker, run_sync_task, load_sync_status_file (pre-existing background-task entry points) - get_deezer_client, parse_youtube_playlist (per-source clients) - get_sync_states (live mutable accessor for the sync UI's state dict) `web_server._register_automation_handlers` now wires those plus the existing infrastructure into a single `AutomationDeps` and calls `register_all`. The 669-line block of closure definitions and engine register calls (lines 959-1627 pre-edit) is gone -- the file shed 743 lines net on this commit. `tests/automation/test_handlers_playlist.py` adds 17 new boundary tests: - discover_playlist: no_id error, specific_id starts worker, all=True enumerates, no playlists in db - refresh_mirrored: error path, source filter (file/beatport excluded), Spotify happy path with auto-discovered marker, per-playlist exception captured into errors counter - sync_playlist: no_id, not_found, no_tracks, no-discovered-tracks skip, discovered-track happy path, unchanged-since-last-sync skip - playlist_pipeline: no_playlist clears running flag, no-refreshable clears running flag, exception clears running flag 3223 tests pass. web_server.py: 35,593 -> 34,850 lines (743 removed).	2 weeks ago
Broque Thomas	ea7d5c65bb	Extract automation handlers (1/N): infrastructure + 3 simple handlers ... Begins the lift of `web_server._register_automation_handlers` (1530 lines, 20 nested closures) into `core/automation/handlers/`. Each extracted handler is a top-level function that accepts `(config, deps)` instead of reaching for module-level globals -- makes them unit-testable in isolation. Infrastructure: - `core/automation/deps.py`: `AutomationDeps` (dependency-injection bundle of clients + callables) and `AutomationState` (mutable flags shared across handler invocations, with thread-safe accessors). - `core/automation/handlers/__init__.py` + `registration.py`: one-stop `register_all(deps)` that wires every extracted handler to the engine. First batch of handlers extracted: - `process_wishlist` -> `core/automation/handlers/process_wishlist.py` - `scan_watchlist` -> `core/automation/handlers/scan_watchlist.py` - `scan_library` -> `core/automation/handlers/scan_library.py` `web_server._register_automation_handlers` now builds the deps once and calls `register_all(deps)` for the extracted batch. Remaining 17 closures still live below; subsequent commits in this branch finish the lift. 14 boundary tests in `tests/automation/test_handlers_simple.py` pin every shape: success path, exception swallow contract, fresh-vs-stale state detection (scan_watchlist's id() trick), guard short-circuits, state cleanup on exceptions, AutomationState concurrent-safe accessors. All 101 automation tests pass; no regression.	2 weeks ago
Broque Thomas	7698405f58	Surface handler-returned errors in automation last_error ... The "Clean Search History" automation card kept showing a stale 'DownloadOrchestrator' object has no attribute 'base_url' error even after the underlying handler bug was fixed in 77d20e9. Root cause is in the engine, not that handler: AutomationEngine only captured uncaught exceptions into last_error. Handlers that report failure by RETURNING {'status': 'error', ...} were treated as successful from the engine's perspective, so subsequent gracefully-failing runs never updated the row to reflect the current state. Both the timer (run_automation) and event (_handle_event_trigger) paths now extract the error string from a result whose status is 'error', falling through 'error' -> 'reason' -> 'message' -> a placeholder so last_error is never None on actual failures regardless of which key the handler chose. Existing behaviour for raised exceptions and successful runs is preserved. Also normalizes _auto_clean_search_history's return key from 'reason' to 'error' so older deployed engines that only check the canonical key still see the failure. Adds 7 regression tests covering every result shape the engine might receive.	1 month ago
Broque Thomas	a8319156ce	Lift /api/automations/blocks static config into core/automation/blocks.py ... The endpoint was returning a 200-line literal dict inline. Moved the three lists (TRIGGERS, ACTIONS, NOTIFICATIONS) to module-level constants in core/automation/blocks.py. Route shrinks to 7 lines. Data is now importable for tests + future docs. Added 8 shape tests so a typo in the dict (missing 'type', wrong field type, missing options on a select, etc.) gets caught by CI instead of breaking the builder UI silently. The `known_signals` field stays computed at request time via _collect_known_signals(database) since it's dynamic. No behavior change. Same response shape. 869 tests passing (was 861). Ruff clean.	1 month ago
Broque Thomas	6cdcf778f3	Lift /api/automations/* into core/automation/ ... Routes moved to thin parse-args/jsonify handlers; logic now lives in three focused modules under core/automation/. 436 lines deleted from web_server.py; 53 added back as wrappers. Module split: - core/automation/api.py — CRUD + run + history helpers. Each function takes (database, automation_engine, ...) explicitly and returns (response_body, http_status). Includes signal cycle detection preflight checks for create + update. - core/automation/progress.py — owns the in-memory progress state dict + lock (mirroring the original web_server.py globals as module-level shared state so all callers see one view), init/update/history helpers, and the WebSocket emit loop. - core/automation/signals.py — collect_known_signals for the builder autocomplete. Out of scope (deferred): - _register_automation_handlers — the 23+ action handler closures stay in web_server.py because each one is tightly coupled to feature- specific implementations (wishlist, watchlist, library scan, etc.). - Worker functions (_process_wishlist_automatically, etc.) — belong with their feature lifts. - _run_sync_task / _run_playlist_discovery_worker — sync + discovery PRs. Behavior preserved 1:1: - Same route response shapes + status codes - Same JSON field hydration (trigger_config, action_config, notify_config, last_result, then_actions) - Same backward-compat: empty then_actions + notify_type set → synthesize then_actions from notify_type/notify_config - Same signal cycle detection behavior on create + update - Same system-automation protection on delete + duplicate - Same reschedule/cancel logic on toggle + bulk-toggle + update - Same progress state shape (status, progress, phase, current_item, log capped at 50, started_at/finished_at, action_type) - Same emit-on-finish socketio push from update_progress - Same emit loop semantics (1s tick, snapshot active states, reap finished after window) Pre-existing bugs preserved (will fix in follow-up PRs): - emit_progress_loop uses naive datetime.now() against tz-aware started_at/finished_at, so the timeout-zombie check raises TypeError → caught → never fires, and the cleanup-after-window check raises → caught → state is reaped on FIRST tick regardless of the window. Tests document this behavior so the next PR can flip them to the corrected expectation. Tests: 72 new under tests/automation/ (signals 10, progress 24, api 38). Full suite: 861 passing (was 789). Ruff clean.	1 month ago