16 Commits (adbdda7b0eeecaaabce5f5b2fa52c026ff84400a)

Author	SHA1	Message	Date
BoulderBadgeDad	c0c4528a28	PR #780 follow-ups: snapshot-based stale check + submit guard + dead code ... - Stale-cache check (playlistTrackCacheIsStale) compared raw track_count to the filtered/cached track list, so any playlist with local or unavailable tracks always looked 'stale' and refetched + re-mirrored on every modal open. Now it compares the upstream snapshot_id (stored at cache time in the shared fetch choke point), and returns not-stale when no snapshot is available — explicit invalidation on refresh still handles real changes. - organize_download: guard executor.submit so a refused job cleans up the batch instead of stranding it in 'analysis' (holding a limited analysis slot). - Removed the dead, deprecated, unused mirrorSpotifyPlaylistTracks.	3 weeks ago
Francesco Durighetto	9ff2e7084a	Fix organize-by-playlist downloads: library entries, wishlist, and stale Spotify cache ... Persist organize_by_playlist on mirrored playlists and run playlist-folder downloads from the auto-sync pipeline instead of the global wishlist phase. Register SoulSync library rows after playlist-folder post-processing, route failed organize batches to the wishlist correctly, and skip sync-time unmatched wishlist only when organize download handles retries. Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL), re-mirror on refetch, and improve standalone playlist modals (re-analysis, Open in Mirrored). Add filesystem missing-track detection and tests. Co-authored-by: Cursor <cursoragent@cursor.com>	3 weeks ago
Broque Thomas	01a867e589	Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes ... Pipeline-driven Auto-Sync runs against any ListenBrainz playlist (Weekly Jams, Weekly Exploration, Top Discoveries, etc.) would sit on ``Refreshing: "<name>"`` with no UI updates for 5-7 minutes before the pipeline progressed. Two real bugs stacked: 1. **Double discovery.** The refresh handler called ``_maybe_discover`` (matching engine, per-track Spotify/iTunes/ Deezer matches) inline for any source returning ``needs_discovery=True`` tracks. Phase 2 of the pipeline then ran the SAME matching engine via ``run_playlist_discovery_worker`` on the same tracks. The refresh-side run blocked the loop with zero progress emission; Phase 2's already has the timed progress-poll pattern. So LB tracks discovered twice, the first time silently. Pipeline now sets ``skip_discovery=True`` on its refresh config. The handler honors the flag and lets Phase 2 handle discovery end-to-end. Standalone callers (Sync-page tab, registration action) leave the flag unset so they still get matched_data on refresh. 2. **No targeted LB refresh.** The LB adapter's ``refresh_playlist`` called ``manager.update_all_playlists()`` — the only refresh entry-point the manager exposed — which re-pulls every cached LB playlist's details from the API (~12+ round-trips) even when only one playlist needed refreshing. Wasteful; tax-on-everyone for one-playlist work. Added ``LBManager.refresh_playlist(mbid)`` — reads the cached playlist_type, fetches just that playlist's details, runs the normal ``_update_playlist`` upsert path. Defaults type to ``user`` for un-cached mbids so new-playlist discovery still works. Skips ``_cleanup_old_playlists`` and ``_ensure_rolling_mirrors_from_cache`` (wasted work for a single-playlist refresh). Also: killed a silent ``except Exception: pass`` in the LB adapter's old refresh wrapper that was masking every LB API failure as a stale-cache hit. Refresh errors now log with full traceback at warning level and propagate ``None`` so the outer handler at ``refresh_mirrored.py:104`` counts the error and surfaces it to the run-history error tally. Pinned with 12 new unit tests across: - ``tests/test_listenbrainz_manager.py`` (8): targeted refresh happy path, unauthenticated guard, empty-mbid guard, upstream ``None`` return, default playlist_type for unknown mbid, exception propagation, cost guard skipping cleanup, skipped- when-unchanged signal - ``tests/test_playlist_sources_adapters.py`` (3): adapter uses targeted call (not legacy), adapter returns ``None`` on manager error (not silent swallow), adapter resolves synthetic series ids before calling the manager - ``tests/automation/test_handlers_playlist.py`` (1): skip_discovery flag bypasses ``_maybe_discover`` end-to-end	1 month ago
Broque Thomas	65d7756da2	Resolve pre-existing ruff lint errors blocking CI ... Five pre-existing lint errors on dev baseline (all introduced May 25-26 before this branch was cut) were blocking CI on this PR. Cleared as courtesy fixes so the merge isn't gated on unrelated tech debt: - web_server.py:22613 — F811 duplicate `urlparse` import inside `_parse_itunes_link_url` (already imported at module top, line 20). Removed from the inline `from urllib.parse import parse_qs, urlparse`; kept `parse_qs` since that one is only used here. - core/listenbrainz_manager.py:746 — S110 silenced with `# noqa: S110 — best-effort lookup, delete proceeds either way`. Matches the existing project convention used in web_server.py:1693, core/watchlist/auto_scan.py:463, core/library_reorganize.py:548. - core/playlists/sources/listenbrainz.py:236 — B905 `zip()` without explicit `strict=`. Added `strict=False` — preserves existing behaviour where `matched` can legitimately be shorter than `match_indices` on partial discover failure. - core/playlists/sources/listenbrainz.py:273 — S110 silenced with `# noqa: S110 — caller falls back to last cached playlist on refresh failure`. - core/playlists/sources/soulsync_discovery.py:105 — S110 silenced with `# noqa: S110 — manager persists last_generation_error on failure; surface existing snapshot`. The existing multi-line comment that already explained the swallow was rolled into the noqa justification so the rule + reason live on one line. Ruff `python -m ruff check .` now passes; 664 discovery + metadata tests still pass.	1 month ago
Broque Thomas	cf5da04439	Roll LB Weekly / Top series into single rolling mirrors (Phase 1c.2.1) ... ListenBrainz publishes "Weekly Jams for X" / "Weekly Exploration for X" with a fresh MBID every week, and "Top Discoveries of YYYY for X" / "Top Missed Recordings of YYYY for X" with a fresh MBID every year. Auto-mirroring those per-period yielded one mirrored- playlist row per week/year — useless for Auto-Sync schedules because the underlying LB playlist never updates, only a brand new playlist replaces it. The user accumulates 100+ dead Weekly Jams rows per year if they discover regularly. This commit collapses each family into a single ROLLING mirror keyed by a synthetic ``source_playlist_id`` (e.g. ``lb_weekly_jams_Nezreka``). Each new period UPSERTs into the same row, so the user gets one stable Auto-Sync schedule per series that automatically picks up the latest period's tracks on every refresh. Non-series LB playlists (user-created, collaborative, Last.fm radios for a specific seed) continue to mirror under their per-playlist MBID as before. Per-period LB playlists are still visible + usable on the LB Sync tab — only the mirror layer collapses. - ``core/playlists/lb_series.py`` (new) — series-detect helper with regex patterns + canonical-name + LIKE-pattern template for each known LB family. Exposes ``detect_series(title)``, ``is_series_synthetic_id(id)``, and ``list_series_synthetic_ids()`` so both the JS auto-mirror hook and the LB adapter can speak the same language. - ``GET /api/listenbrainz/series-detect?title=...`` — thin HTTP shim around ``detect_series`` so the auto-mirror JS doesn't duplicate the regex. - ``ListenBrainzPlaylistSource.get_playlist`` now recognizes synthetic series ids — it queries the LB cache for the newest cache row whose title matches the series' LIKE pattern and resolves to that row's MBID before fetching tracks. The mirror's meta keeps the synthetic id so refreshes always re-resolve to the latest period. - ``_mirrorListenBrainzAfterDiscovery`` (sync-services.js) calls the new detect endpoint when discovery completes — if a match comes back it swaps the per-period MBID for the synthetic id + the canonical name. Existing Last.fm radio routing logic stays intact (Last.fm radios aren't a series). - ``ListenBrainzManager._cleanup_per_period_series_mirrors`` — one-shot consolidation sweeper runs in ``_cleanup_old_playlists`` + deletes any legacy per-period mirror rows so the consolidated rolling mirror is the only one left. Idempotent — only matches per-period titles ("Weekly Jams for ..., week of ...") and never the canonical rolling-mirror titles ("ListenBrainz Weekly Jams"). - 11 new tests pin the detector + synthetic-id helpers; 236 total across adapter + automation + lb-series suites green.	1 month ago
Broque Thomas	246503066b	Fold provider-matching into PlaylistSource contract (Phase 1b) ... Adds ``discover_tracks(tracks) -> List[NormalizedTrack]`` to the PlaylistSource interface. Sources whose tracks already carry provider IDs (Spotify, Tidal, Qobuz, YouTube, Deezer, Spotify public, iTunes link, SoulSync Discovery) inherit a no-op default; ListenBrainz + Last.fm override to run the matching engine. This closes the last gap before LB / Last.fm / SoulSync Discovery can land as Sync-page mirror sources: the refresh handler now calls ``source.discover_tracks(...)`` whenever a source returns tracks with ``needs_discovery=True``, so mirrored LB rows arrive already discovered + ready for the sync pipeline. Previously, LB playlists ran through a separate state-machine worker tied to the Discover-page UI, with results stored in ``discovery_cache`` instead of ``mirrored_playlist_tracks.extra_data``. Changes: - ``core/playlists/sources/base.py`` — PlaylistSource switches from Protocol to ABC so a concrete default for ``discover_tracks`` can live on the base class. The four real-work methods stay ``@abstractmethod``; instantiating an adapter that forgets one fails loudly at construction. - ``core/discovery/matching.py`` (new) — pure ``match_mb_tracks`` helper that runs Strategy-1-only matching-engine queries against Spotify (primary) or iTunes (fallback). No state machine, no discovery-cache writes, no wing-it stub — that richer flow stays in ``core/discovery/listenbrainz.py`` for the Discover-page UI. - ``ListenBrainzPlaylistSource`` + ``LastFMPlaylistSource`` take an optional ``discover_callable`` constructor arg. Last.fm reuses the LB implementation since the track shape is identical. - ``bootstrap.build_playlist_source_registry`` accepts a ``discover_callable`` kwarg and wires it into LB + Last.fm adapters. - ``web_server.py`` boot constructs the discovery callable from the existing matching engine + ``_discovery_score_candidates`` + Spotify / iTunes clients, passes through to the registry. - ``refresh_mirrored.py`` adds a small ``_maybe_discover`` helper that calls ``source.discover_tracks(...)`` between fetch and ``to_mirror_track_dict`` projection — only fires when at least one track has ``needs_discovery=True``, so the normal Spotify / Tidal / etc. refresh path stays a zero-cost pass-through. Tests: - 5 new adapter tests: default no-op pass-through, LB discovery with mixed matches/misses, LB no-callable fallback, Last.fm shares the LB implementation, mirror-dict spotify_hint emit. - 1 new automation test: end-to-end LB refresh with a stub discover_callable proves the matched_data lands in ``mirror_playlist_tracks.extra_data`` after the registry refresh + discover hop. 225 tests across adapter + automation suites green.	1 month ago
Broque Thomas	8c41b05fe8	Refactor refresh_mirrored to use unified PlaylistSource registry ... Phase 1a of the Discover-to-Sync unification. The mirrored-playlist refresh handler used to branch per-source through a ~190-line if/elif chain (Spotify, Spotify public, Deezer, Tidal, YouTube). Each branch hand-built its own ``extra_data`` JSON for the matched- data block. With every new source we considered for Sync-page mirror support (ListenBrainz, Last.fm radio, SoulSync Discovery, iTunes link), that chain would have grown a new elif. This commit lifts the per-source logic into the existing adapter layer and collapses the dispatch to a registry lookup: - ``core/playlists/sources/deezer.py`` — new adapter so the registry covers every source the refresh handler previously branched on. - ``core/playlists/sources/bootstrap.py`` — single helper that builds a populated registry from injected getter callables. Both ``web_server.py`` boot and the automation test fixtures call it, so the two construction paths can't drift. - ``core/playlists/sources/base.py`` — ``to_mirror_track_dict`` projection helper centralises the NormalizedTrack → DB-row conversion (including the discovered/matched_data and spotify_hint extra_data shapes the downstream sync + wishlist consumers already expect). - Spotify adapter now populates ``extra['discovered']`` + an ``extra['matched_data']`` block when fetching via the authed API, so Spotify mirrors keep landing pre-discovered (matches the pre-refactor contract pinned by ``test_spotify_refresh_writes_to_db``). - Spotify-public adapter populates ``extra['spotify_hint']`` so the discovery worker can skip its search step and jump straight to enrichment for the known track ID. - All artist-name fields now project to first-artist-only across every adapter — matches the pre-refactor mirror_playlist DB shape (``t.artists[0]``). ``refresh_mirrored.py`` shrinks ~190 → ~80 lines and keeps: - the file/beatport unrefreshable-source filter, - URL extraction from ``description`` via ``require_refresh_url`` for spotify_public + youtube, - the Spotify-public → authed-Spotify fallback when the user is signed in (handler-level branch, not in any adapter), - the Tidal-not-authenticated soft-skip log (skip, not error), - existing-extra_data preservation across refreshes, - the ``playlist_changed`` automation event emit on track-set delta. Test scaffolding: - ``_build_deps`` in ``tests/automation/test_handlers_playlist.py`` now builds a default registry from the passed clients via ``build_playlist_source_registry``, so existing refresh tests exercise the same path without per-test changes. New tests cover Tidal-not-authed soft-skip, Deezer refresh writes plain tracks, YouTube refresh reads URL from description, and Spotify-public uses authed Spotify when signed in. - 4 new adapter tests for Deezer projection + ``to_mirror_track_dict`` (minimal track, Spotify matched_data, Spotify-public spotify_hint). - ``playlist_source_registry`` field on ``AutomationDeps`` defaults to ``None`` so the other 5 automation test files (which don't exercise refresh_mirrored) keep working unchanged. 220 tests across automation + adapter suites green.	1 month ago
Broque Thomas	c5898c3b9b	Add unified PlaylistSource adapter layer (Phase 0) ... Groundwork for unifying Discover-page playlists (ListenBrainz, Last.fm radio, SoulSync Discovery) with Sync-page playlists (Spotify, Tidal, Qobuz, YouTube, Spotify public, iTunes link). All nine sources now expose the same `PlaylistSource` Protocol so callers stop having to branch per-source. This commit only adds the abstraction — no dispatch sites collapse to the registry yet, no DB or UI changes. Adapters wrap existing clients via injected getter callables to avoid eager imports of web_server.py globals. - core/playlists/sources/base.py — PlaylistMeta, NormalizedTrack, PlaylistDetail dataclasses + PlaylistSource Protocol with supports_listing / supports_refresh / requires_auth capability flags. needs_discovery flag on NormalizedTrack marks tracks that carry raw MB metadata (LB, Last.fm) vs tracks already matched to a provider ID (everything else). - core/playlists/sources/registry.py — thread-safe lazy-factory registry with instance caching + re-register invalidation. - nine adapters in core/playlists/sources/ wrapping SpotifyClient, TidalClient, QobuzClient, spotify_public_scraper, the YouTube + iTunes-link parsers (via injected callables), ListenBrainzManager, Last.fm radio rows in the ListenBrainz cache, and PersonalizedPlaylistManager. - tests/test_playlist_sources_adapters.py — 18 tests covering each adapter's field projection with fake backing clients, plus registry lazy-construct + cache + re-register invalidation. Phase 1 will collapse refresh_mirrored.py's per-source if/elif chain to a registry lookup and surface ListenBrainz as a Sync-page tab.	1 month ago
Broque Thomas	efdcde1892	Add playlist auto-sync run history ... Persist per-playlist pipeline run snapshots from the shared playlist pipeline, expose a history API, and upgrade the Auto-Sync modal with live pipeline monitoring, Run now controls, and a runs-style history tab.	1 month ago
Broque Thomas	f83c671570	Add direct mirrored playlist pipeline runs ... Expose playlist-native run and status endpoints that reuse the shared mirrored playlist pipeline engine while routing progress into playlist UI state. Add a Run Pipeline action to mirrored playlist cards and modals with live status polling, and make the shared pipeline lock atomic for manual and scheduled callers.	1 month ago
Broque Thomas	bc6bacb7da	Move mirrored playlist pipeline into playlist domain ... Extract the all-in-one mirrored playlist lifecycle into core/playlists/pipeline.py so automation becomes a thin adapter. Preserve the existing automation action and behavior while making the pipeline reusable by future direct playlist UI controls.	1 month ago
Broque Thomas	547e499121	Expose mirrored playlist source-ref health ... Return normalized source_ref metadata from mirrored playlist APIs so the UI no longer has to infer editable refresh links from description fields. Accept Spotify embed URLs during source-ref repair and add coverage for source-ref health reporting.	1 month ago
Broque Thomas	73bd2db547	Harden playlist pipeline source refresh ... Centralize mirrored playlist source reference normalization so edited links and IDs are stored consistently. Preserve URL-backed refresh refs, surface missing-source refresh failures, count background sync failures in pipeline summaries, and retry guarded automation skips after a short delay instead of losing a scheduled run. Add focused coverage for source refs, mirrored playlist source updates, refresh failures, and guarded retry behavior.	1 month ago
Broque Thomas	9602d1827c	Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites ... Catches the silent excepts the awk-based earlier sweeps missed: - Bare `except:` followed by `pass` (also swallows KeyboardInterrupt and SystemExit — actively wrong). Upgraded to `except Exception as e: logger.debug("...: %s", e)`. ~14 sites across connection_detect, soulseek_client, listenbrainz_manager, watchlist_scanner, youtube_client, navidrome_client, jellyfin_client, web_server. - `except Exception:` + pass that the awk pattern missed (e.g. multi-line or unusual whitespace). ~31 sites across automation_engine, database_update_worker, music_database, spotify_client, web_server, others. - 14 legitimate cleanup sites left silent with explicit `# noqa: S110` + comment explaining why (atexit handlers, finally-block conn.close calls). Logging during shutdown can itself crash because file handles get torn down before the handler fires. Also enables `S110` rule in pyproject.toml so this pattern fails CI going forward — drift fails at PR review instead of at runtime against a wedged worker thread. Tests path keeps S110 ignored (test fixtures legitimately use try-except-pass for cleanup). Adds a WHATS_NEW entry to helper.js summarizing the full #369 sweep. Verified: `python -m ruff check .` → All checks passed. Verified: `python -m pytest tests/` → 2188 passed. Closes #369	2 months ago
Broque Thomas	aa54bed818	Surface silent exceptions across remaining modules — ~70 sites ... Final sweep. Covers: - Downloads: candidates / lifecycle / master / monitor / wishlist_failed - Metadata: source / registry / cache / common / artwork (+ plex_client) - Imports: pipeline / resolution / file_ops / paths / guards - Library: path_resolver / retag / duplicate_cleaner - Stats / playlists / wishlist / discovery / automation / enrichment - Misc: hydrabase_client, soulsync_client, tag_writer, debug_info, api_call_tracker, album_consistency, beatport_unified_scraper, reorganize_runner, seasonal_discovery, lidarr_download_client, services/sync_service.py, automation_engine, automation/progress Two `_e` renames in imports/file_ops.py (outer scope binding `e`). A few finally-block sites in metadata/album_mbid_cache.py, library/track_identity.py, listening_stats_worker.py, watchlist/ auto_scan.py left silent — same reason as the rest of the sweep (logger calls during cleanup paths can themselves raise). Refs #369	2 months ago
Broque Thomas	5c8b8b271a	Lift _prepare_stream_task + playlist_explorer_build_tree to core/ ... Final lift in the web_server.py extraction effort. Pulls two route handlers + one background worker out of `web_server.py` into new focused packages: - `core/streaming/prepare.py` — 258-line stream-prep worker that downloads a track to the local Stream/ folder for the browser audio player. - `core/playlists/explorer.py` — 305-line route handler for `POST /api/playlist-explorer/build-tree` that streams an NDJSON discography tree from a mirrored playlist. What `prepare_stream_task` does: 1. Reset stream state to 'loading' with the new track info. 2. Clear any prior file from Stream/ (only one stream lives there). 3. Spin up a fresh asyncio event loop and `soulseek_client.download()`. 4. Poll progress every 1.5s. Queue timeout 15s; overall 60s. 5. On succeeded + bytes-match: find the file with retry, move into Stream/, signal slskd completion, mark state 'ready' with file_path. 6. On error/timeout/cancel: state goes to 'error' or 'stopped'. 7. Finally: tear down the event loop cleanly. What `playlist_explorer_build_tree` does: 1. Validate request, load playlist + tracks from DB. 2. Pick active metadata source (Spotify if authed, else fallback). 3. Group tracks by artist using discovered matched_data when the provider matches the active source. 4. Stream NDJSON: meta line → one artist line per group → complete line. 5. Per artist: cache check → resolve discography → tag releases with `in_playlist` flag based on title-similarity match → filter by mode (`albums` = only matches; `discographies` = full disco). 6. Mark playlist as explored on completion. Strict 1:1 byte parity: Both functions exposed their dependencies through proxy patterns established in earlier lifts (PR4–PR8). For prepare_stream_task, `stream_state` is a deps property; for the explorer, Flask `request` / `jsonify` / `Response` are injected via deps so the lifted body keeps its native syntax. Both lifts verified ZERO diff against the original after `deps.X` → global X normalization. 258 lines orig = 258 lines lifted (prepare_stream_task). 305 lines orig = 305 lines lifted (explorer). Bonus cleanup: web_server.py's module-level `import shutil` and `import glob` were now unused (only `_prepare_stream_task` used them at module scope; every other reference is via inline `import shutil` in respective function bodies). Removed both module-level imports — ruff caught the F811 redefinitions and confirmed they're truly redundant. Dependencies for `PrepareStreamDeps` (11 fields): config_manager, soulseek_client, stream_lock, project_root, docker_resolve_path, find_streaming_download_in_all_downloads, find_downloaded_file, extract_filename, cleanup_empty_directories, plus 2 stream_state property delegates. Dependencies for `PlaylistExplorerDeps` (9 fields): Flask request/Response/jsonify, spotify_client, get_database, get_active_discovery_source, get_metadata_fallback_client, get_metadata_fallback_source, get_metadata_cache. Tests: 6 new under tests/streaming/test_prepare.py (state init, Stream/ folder creation + clearing, download-init failure, completed + moved + ready state, partial-bytes incomplete-warning path) plus 9 new under tests/playlists/test_explorer.py (5 validation early-exit paths, streaming response shape with meta/complete lines, mark- explored side effect, discovered-artist grouping using matched_data, provider mismatch falling back to raw artist name). Full suite: 1355 passing (was 1340). Ruff clean. End of the web_server.py extraction effort. Started at ~45,000 lines across PR4–PR8 + this commit; finished around 35,000 lines with the heavy worker + route logic now living in domain-cohesive packages under core/. The remaining bulk in web_server.py is route handlers, service initialization, and the deferred 1530-line `_register_automation_handlers` (startup-only, marginal lift value).	2 months ago