7 Commits (dev)

Author	SHA1	Message	Date
Broque Thomas	877d0e7d81	Personalized pipeline: auto-refresh stale snapshots after watchlist scan ... Snapshots now track when their source data changes. Watchlist scan emits stale flags on the playlists whose underlying pool just got refreshed; the next pipeline run sees the flag and regenerates the snapshot before syncing, so the server playlist never lags the source. Schema: - new `is_stale INTEGER NOT NULL DEFAULT 0` column on `personalized_playlists`, plus an idempotent ADD COLUMN migration in `ensure_personalized_schema` for installs created before this PR. - `PlaylistRecord.is_stale: bool = False` exposed on the dataclass so callers can branch on freshness without re-querying. Manager: - new `mark_kinds_stale(kinds, profile_id=None)` flips the flag in bulk for a list of kinds (used by upstream data refreshers). - `_persist_snapshot` clears `is_stale = 0` on successful refresh. - SELECT statements + `_row_to_record` updated to read the column (with tuple-form length guard for safety). Pipeline: - `_build_payloads_for_kinds` now branches: refresh_first=True OR `existing.is_stale` -> refresh_playlist, else read existing snapshot. So the auto-refresh kicks in without needing the user to toggle the refresh-each-run option. Watchlist scanner emits stale flags at three sites: - after `update_discovery_pool_timestamp` -> marks pool-fed kinds stale: hidden_gems, discovery_shuffle, popular_picks, time_machine, genre_playlist, daily_mix. - after release_radar `save_curated_playlist` -> marks `fresh_tape`. - after discovery_weekly `save_curated_playlist` -> marks `archives`. All three calls go through a module-level `_mark_personalized_kinds_stale` helper that builds a PersonalizedPlaylistManager with `deps=None` (only DB access is needed for the flag update — no generator dispatch). Each call is wrapped in try/except so a flag failure can never abort the scan itself. Tests: - new `TestStaleFlag` class in `test_personalized_manager.py` (6 tests): default-false, single-kind flip, multi-kind, profile scoping, refresh-clears, empty-list noop. - two new pipeline tests pin the auto-refresh dispatch: `test_stale_snapshot_auto_refreshes_even_without_refresh_first` and `test_non_stale_snapshot_skips_refresh`. - existing stub-manager `SimpleNamespace` returns gained `is_stale=False` so the new attribute read doesn't AttributeError. Full suite: 3391 pass. User-facing WHATS_NEW entry added under 2.5.2 (above the prior pipeline auto-sync entry) describing the auto-refresh behavior.	1 week ago
Broque Thomas	e1f0810df5	Personalized pipeline: UI multi-select picker for kinds + variants ... The action was registered + the block declared, but the automation builder's per-action config renderer didn't have a case for `personalized_pipeline` so users only saw the bare card with the generic delay-minutes input — no way to select which playlists to sync. This commit adds the multi-select picker. Backend: - `core/personalized/api.list_kinds(manager=...)` now optionally takes a manager and includes the resolved variant list per kind (calls each spec's variant_resolver(deps) when present). Singleton kinds get an empty `variants` list. Variant-bearing kinds (time_machine / genre_playlist / daily_mix / seasonal_mix) get their full enumerated set. - `web_server.py` `/api/personalized/kinds` route now passes a built manager so the variants list lands in the response. Frontend: - `webui/static/stats-automations.js` `_renderBlockConfigFields` gains a `personalized_pipeline` branch that renders a scrollable multi-select picker: - Singletons (Hidden Gems, Discovery Shuffle, Popular Picks, Fresh Tape, The Archives) = one checkbox row per kind - Variant kinds = a section header + one checkbox row per variant (e.g. Time Machine: 1960s/1970s/.../2020s; Seasonal: halloween/ christmas/valentines/summer/spring/autumn) - Pre-checks rows that match the existing `kinds` config on edit - New `_autoLoadPersonalizedKinds(slotKey)` fetches `/api/personalized/kinds` (cached after first load), renders the picker DOM, and pre-checks saved selections via `data-kind` / `data-variant` attributes on the checkboxes. - `_renderBuilderCanvas` calls the loader for any `cfg-*-kinds-picker` it finds in the freshly-rendered slots. - The save-time `_collectActionConfig` walks the picker's checked inputs (matched by `data-kind` attribute) and emits `{kinds: [{kind, variant?}, ...], refresh_first, skip_wishlist}` in the same shape the handler expects. Tests: - `tests/automation/test_automation_blocks.py::_FIELD_TYPES` adds 'personalized_playlist_select' so the block-shape regression test accepts the new field type. (Test was failing because it whitelists every field type used across all blocks.) - 189 automation + personalized API tests pass; full suite intact.	1 week ago
Broque Thomas	3f965f48cd	Personalized playlists: ruff B905 — explicit strict= on zip() ... CI ruff check failed on the seasonal_mix tuple-row coercion path where a `zip(columns, row)` call lacked an explicit `strict=`. Set `strict=False` to preserve the original intent (tolerant if the row shape ever drifts from the column tuple). The SELECT always returns 8 columns so the lengths match in practice; using strict=False just avoids a future raise if a generator drift changes that. Live happy path stays unchanged: rows from sqlite3.Row hit the `hasattr(r, 'keys')` branch above and never reach the zip line. The zip branch only runs for plain-tuple rows in tests.	1 week ago
Broque Thomas	cc0828e9ff	Personalized playlists (4/N): staleness post-filter (exclude_recent_days) ... Adds the first quality feature on top of the manager: when `config.exclude_recent_days > 0`, the manager drops any track from the generator's output whose primary id was served by this kind for this profile in the last N days. Lives at the manager layer, not in each generator, so: - generators stay focused on selection logic - staleness behavior stays uniform across every kind - enabling/disabling per playlist is just a config patch Implementation: - New `PersonalizedPlaylistManager._apply_quality_filters` runs after generator returns, before `_persist_snapshot`. - Reads recent ids via existing `recent_track_ids` accessor. - Tracks without a primary id pass through unchanged (nothing to dedupe on -- happens for sourceless tracks during edge cases). - Returns a new list (never mutates input). Default `exclude_recent_days = 0` preserves pre-overhaul behavior. Per-playlist override via `PUT /api/personalized/playlist/<kind>/config` with `{"exclude_recent_days": N}`. Recommended values: - Discovery Shuffle: 1-3 days (high churn desired) - Hidden Gems: 7-14 days (avoid same gems weekly) - Time Machine / Genre: 30+ days (slow rotation OK, stable view preferred) 4 new boundary tests: - Zero days = no filter (default behavior preserved) - Positive days drops tracks served in window - Filter preserves new tracks alongside dropped ones - Tracks without primary id pass through unchanged 3369 tests pass total. Note: listening-history cross-ref + seeded shuffle are deferred to a future PR. Each requires deeper integration -- listening history needs a play-events table the discovery pool can query against; seeded shuffle needs the legacy generators to accept a seed param without breaking their existing diversity / popularity logic.	1 week ago
Broque Thomas	9f383acbfb	Personalized playlists (3/N): standardized API endpoints ... Wraps the manager + generator dispatch behind one HTTP surface so the UI can drop the patchwork `/api/discover/personalized/*` calls in favor of a single REST shape. Legacy endpoints stay alive for backward compat during the UI migration window. New endpoints: - GET /api/personalized/kinds — list every registered kind + metadata - GET /api/personalized/playlists — list every persisted playlist for the active profile - GET /api/personalized/playlist/<kind> — fetch singleton + tracks - GET /api/personalized/playlist/<kind>/<variant> — fetch variant + tracks - POST /api/personalized/playlist/<kind>/refresh — regenerate singleton - POST /api/personalized/playlist/<kind>/<variant>/refresh — regenerate variant - PUT /api/personalized/playlist/<kind>/config — patch singleton config - PUT /api/personalized/playlist/<kind>/<variant>/config — patch variant config Per-call manager construction wires the deps each generator needs: - database (MusicDatabase singleton) - service (PersonalizedPlaylistsService for legacy generator calls) - seasonal_service (SeasonalDiscoveryService for seasonal_mix) - get_current_profile_id (active profile accessor) - get_active_discovery_source (source dispatcher) API handlers themselves live as pure functions in `core/personalized/api.py` so they're testable without Flask. The Flask layer in `web_server.py` is a thin parse-body / call-handler / jsonify wrapper. 11 new boundary tests (122 personalized total): - list_kinds enumerates registry, exposes default config + tags - list_playlists returns empty list when none exist, serializes PlaylistRecord shape correctly - get_playlist_with_tracks auto-creates on first access, returns persisted tracks, raises ValueError on unknown kind - refresh_playlist runs generator and returns track snapshot, forwards config_overrides to the generator - update_config patches stored config 3365 tests pass total. Manager construction triggers generator registration via `from core.personalized import generators` import side-effect.	1 week ago
Broque Thomas	53284ee7c8	Personalized playlists (2/N): all 8 generators wired through manager ... Adds the per-kind generator modules and registers them with the PlaylistKindRegistry so the manager's `refresh_playlist` can dispatch to any of them. Generators (each in its own module under `core/personalized/generators/`): Singletons (variant=''): - hidden_gems -> wraps service.get_hidden_gems - discovery_shuffle -> wraps service.get_discovery_shuffle - popular_picks -> wraps service.get_popular_picks Variant-bearing kinds: - time_machine -> variant = decade label ('1980s', '1990s', ...). Variant resolver returns 7 standard decades. Generator parses '1980s' -> 1980 + delegates to service.get_decade_playlist. - genre_playlist -> variant = URL-safe genre key ('electronic_dance', 'hip_hop_rap', ...). Resolver normalizes parent-genre keys from service.GENRE_MAPPING; free-form keywords pass through to service.get_genre_playlist. - daily_mix -> variant = top-genre rank ('1' / '2' / '3' / '4'). Generator looks up user's Nth-ranked library genre and returns discovery picks within it. Library half (was a stub returning []) is intentionally dropped: tracks table has no source IDs, so library rows can't sync. Fixed the stub to return [] cleanly without the misleading log warning. - fresh_tape -> Spotify Release Radar. Reads curated track IDs from discovery_curated_playlists (tries 'release_radar_<source>' first, falls back to 'release_radar') and hydrates against the discovery pool. - archives -> Spotify Discover Weekly. Same hydration path as fresh_tape but uses 'discovery_weekly'. - seasonal_mix -> variant = season key ('halloween' / 'christmas' / 'valentines' / 'summer' / 'spring' / 'autumn'). Reads curated IDs via SeasonalDiscoveryService then hydrates from seasonal_tracks (which carries full track_data_json). Each module: - Defines `generate(deps, variant, config) -> List[Track]`. - Defines `SPEC = PlaylistKindSpec(...)` and registers it on import (idempotent — re-import safe via `if registry.get(...) is None`). - For variant-bearing kinds, also defines `variant_resolver(deps)`. Shared helpers in `_common.py`: - `get_service(deps)` pulls the legacy `PersonalizedPlaylistsService` instance (deps.service or deps['service']). - `coerce_tracks(rows)` runs each dict through `Track.from_dict`, tolerates None / non-list inputs. Tests (50 new, total 85 across personalized subsystem): - Singletons: registration + display name + dispatch + limit forwarding + empty/None tolerance + missing-deps error + dict-form deps acceptance (16 tests). - Variants: variant_resolver listing + label parsing + invalid variant errors + parent-key normalization + free-form passthrough (13 tests). - Curated/hybrid: daily_mix rank-to-genre resolution + rank-out-of- range empty + invalid-variant error; fresh_tape & archives hydration order + missing-id skip + source-specific-then-fallback key dispatch + limit + missing-database-dep error; seasonal_mix curated-id hydration order + missing-id skip + JSON round-trip + empty-curated empty + limit + missing-service error (21 tests). 3304+ tests pass. No regression on existing 62 personalized tests.	1 week ago
Broque Thomas	79224ed294	Personalized playlists (1/N): unified storage + manager foundation ... Begins the standardization of the personalized-playlist subsystem. Pre-existing state was a patchwork: Group A (Fresh Tape / Archives / Seasonal Mix) lived in `discovery_curated_playlists` and `curated_seasonal_playlists` with inconsistent shapes; Group B (Hidden Gems / Discovery Shuffle / Time Machine / Popular Picks / Genre / Daily Mixes) was computed on-demand by `PersonalizedPlaylistsService` with no persistence -- every call reran the generator with `ORDER BY RANDOM()` so results rotated. Post-overhaul (this PR) every personalized playlist lands in one unified storage layer with stable identity, persistent track lists, explicit refresh, and per-playlist user-tweakable config. Foundation in this commit (no behavior change yet): - `database/personalized_schema.py`: 3 tables created idempotently at app startup (wired into `MusicDatabase._initialize_database`). - `personalized_playlists`: one row per (profile, kind, variant) with config_json, track_count, last_generated_at, last_synced_at, last_generation_source, last_generation_error. Variant '' (empty string) for singletons; non-empty for time_machine / seasonal_mix / genre_playlist / daily_mix. - `personalized_playlist_tracks`: current snapshot per playlist. Atomically replaced on refresh. - `personalized_track_history`: append-only log powering the `exclude_recent_days` config knob. - `core/personalized/types.py`: `Track`, `PlaylistConfig`, `PlaylistRecord` dataclasses. `PlaylistConfig.merged()` for partial-update PATCH semantics; `Track.from_dict()` accepts the legacy generator output shape unchanged. - `core/personalized/specs.py`: `PlaylistKindSpec` (kind, name_template, default_config, generator, variant_resolver) and a module-level registry. Generators register at import time; manager dispatches by kind. - `core/personalized/manager.py`: `PersonalizedPlaylistManager` -- the only thing that touches the new tables. Owns: - ensure_playlist (auto-create row from kind defaults) - get_playlist / list_playlists - refresh_playlist (atomic snapshot replace; generator exception preserves previous good snapshot + records error on row) - get_playlist_tracks - update_config (deep-merge with stored config, including extra dict) - recent_track_ids (staleness lookup for generators) 35 boundary tests in `tests/test_personalized_manager.py` pin every shape: config round-trip / merge semantics / extra deep-merge / defaults; Track.from_dict tolerance + primary_id fallback chain; registry dedup / display_name with+without variant; manager ensure_playlist auto-create + idempotency, variant separation, required-variant enforcement, unknown-kind error; refresh persists + replaces atomically + survives generator exception with previous snapshot intact + records source from first track + round-trips nested track_data_json; update_config patch semantics; list_playlists profile scoping; staleness history scoped to (profile, kind, days). 3304 tests pass total. Generators ship in subsequent commits on this branch -- each kind migrated one at a time with its own per-kind boundary tests.	1 week ago