SoulSync

Commit Graph

Author	SHA1	Message	Date
Broque Thomas	6a75d656fa	Cin-2: Generic accessors on orchestrator + singleton factory Cin's review feedback: external callers reach per-source clients via attribute access (orch.hifi.reload_instances()) — needs generic accessors so the registry IS the single source of truth. Adds: - orch.client(name) — public accessor for a per-source client. Resolves canonical names (deezer) AND legacy aliases (deezer_dl). - orch.configured_clients() — returns {name: client} for every initialized AND is_configured() == True source. Replaces the 6+ if/hasattr/is_configured chain Cin called out: if hasattr(orch, 'soulseek') and orch.soulseek and \ orch.soulseek.is_configured(): ... - orch.reload_instances(source=None) — generic dispatch for source-specific reload calls. Replaces orch.hifi.reload_instances() with orch.reload_instances('hifi'). - get_download_orchestrator() / set_download_orchestrator() singleton factory matching Cin's get_metadata_engine pattern in PR #498. web_server.py can install the orchestrator it builds at boot so future callers grab via the factory instead of importing the legacy `soulseek_client` global. Phase Cin-3/Cin-4 will replace existing call sites; this commit just provides the surface so those migrations are mechanical. Suite still green (335 download tests + 6 new generic-accessor tests).	4 weeks ago
Broque Thomas	0ee092979e	Self-review fixes before opening PR Three findings from a final review pass: 1. Worker clobbered Cancelled with Errored when impl returned None / raised mid-cancel. The legacy per-client thread workers each had a guard (``if state != 'Cancelled': state = 'Errored'``); the shared worker dropped it. Fix: new ``_mark_terminal`` helper in BackgroundDownloadWorker reads current state before writing the terminal one and leaves Cancelled alone. SoundCloud test updated back to the strict Cancelled-only assertion (had been loosened to accept Errored as a workaround). Two new pinning tests catch the regression. 2. Dead code in engine.py. ``find_record`` and ``iter_all_records`` had no production callers — only tests. Removed them. Concurrent-add stress test rewritten to use the per-source iterator that's actually in use. 3. Silent ``except Exception: pass`` in cross-source query methods. Faithful to legacy behavior (one source failing shouldn't take down aggregation) but Cin's standard is "log even when you swallow." Each silent-swallow site now logs at debug level so the source name + exception are inspectable without adding warning-level noise. Suite still green (2049 passed).	4 weeks ago
Broque Thomas	4b4076b57f	F: Engine owns hybrid fallback chain `engine.search_with_fallback(query, source_chain, ...)` walks the chain in order, skips unconfigured / unregistered plugins, swallows per-source exceptions, and returns the first non-empty (tracks, albums) tuple. Replaces orchestrator's hand-rolled hybrid search loop. `engine.download_with_fallback(username, filename, file_size, source_chain)` falls through the chain when a source returns None / raises. Username hint promotes a matching source-chain entry to head of order. NOT yet wired into orchestrator.download — today's username comes from a search result and represents the user's explicit source pick, so silently falling through would override their choice. Engine method is available for future callers that want fallback semantics (search_and_download_best, automation). Orchestrator gains _resolve_source_chain helper that builds the ordered list (hybrid_order config, falling back to legacy primary/secondary pair). Orchestrator.search hands chain off to engine.search_with_fallback for hybrid mode. 8 new tests pin the fallback semantics: chain ordering, unconfigured-skip, exception-continue, empty-when-exhausted, username-hint promotion. Suite still green (2050 passed).	4 weeks ago
Broque Thomas	1062589501	E2: Migrate YouTube to declared rate-limit policy YouTubeClient gains rate_limit_policy() that returns a RateLimitPolicy with the configured download_delay (3s default from `youtube.download_delay`). Engine reads this at register_plugin time + applies to engine.worker. set_engine still re-applies the delay so runtime reload_settings updates flow through the same pathway. Other sources keep the default policy (concurrency=1, delay=0) which matches their current behavior — no migration needed beyond YouTube which is the only source with a non-default download throttle today. New pinning test asserts the policy shape (delay=3.0, concurrency=1). Suite still green (2042 passed).	4 weeks ago
Broque Thomas	a3929b457b	E1: Add RateLimitPolicy declaration mechanism `core/download_engine/rate_limit.py` introduces a per-source policy declaration: download_concurrency + download_delay_seconds. Plugins declare via `RATE_LIMIT_POLICY` class attribute or a `rate_limit_policy()` method. Engine applies the declared policy to engine.worker at register_plugin time — set_concurrency + set_delay get pushed in automatically. Plugins without a declaration get the conservative default (1 / 0). The set_engine callback fires AFTER policy registration so config-driven sources (YouTube reads user-tunable youtube.download_delay) can override. Plan doc updated to reflect Phase D skip (search code is 90% source-specific, not 60% — lifting it would be lossy or bloated). Pure additive — no plugin migrated yet. 8 tests pin the resolution priority + engine wire-up + override semantics. Suite still green (327 download tests).	4 weeks ago
Broque Thomas	fdb3e44965	C7: Migrate SoundCloud to engine.worker Last C-phase migration. Same pattern as C2-C6 — SoundCloud drops active_downloads + _download_lock + _download_thread_worker. download() delegates to engine.worker.dispatch with permalink_url captured in a closure so the impl gets the URL (not the track_id) yt-dlp needs. Both progress hooks (HLS-fragmented + byte-based) write to engine state via update_record. Query/cancel methods read engine state. Existing test_soundcloud_client.py mass-updated: 16 tests that reached into client.active_downloads / _download_lock now use engine.add_record / get_record / update_record via a small _wire_engine helper. test_download_thread_does_not_clobber_cancelled_state now accepts either Cancelled or Errored as the final state since the engine.worker doesn't preserve Cancelled-over-Errored the way the legacy per-client thread did (potential follow-up: add that guard uniformly in BackgroundDownloadWorker). Phase A pinning tests updated. Suite still green (2033 passed).	4 weeks ago
Broque Thomas	cf438ba2d6	C6: Migrate Deezer to engine.worker Same migration pattern as C2-C5. Deezer-specific quirks preserved through worker overrides: - username_override='deezer_dl' (legacy slot frontend reads) - thread_name='deezer-dl-<track_id>' (diagnostic naming) - track_id stays as STRING (Deezer GW API uses string IDs) - Extra 'error' slot in record for ARL re-auth failure messages Mid-download chunk loop's many state mutations (cancellation checks, progress updates, error capture across multiple failure modes) all flow through engine.update_record / get_record now. Added _set_error and _is_cancelled helpers to keep call sites readable. Pinning tests updated. Suite still green (319 download tests).	4 weeks ago
Broque Thomas	27a97f8af6	C5: Migrate HiFi to engine.worker Same pattern as C2/C3/C4. HiFi worker was named _download_worker (not _thread_worker like the others) — gone now along with the state dict + lock. Mid-download HLS-segment progress hook (_update_download_progress) writes to engine state. Pinning tests updated. Suite still green (318 download tests).	4 weeks ago
Broque Thomas	7944568c5c	C4: Migrate Qobuz to engine.worker Same migration pattern as C2/C3. QobuzClient drops active_downloads + _download_lock + _download_thread_worker. Mid-download chunk-progress mutations + cancel-state checks flow through engine.update_record / get_record. Query/cancel methods read engine state. Pinning tests updated to assert engine state. Suite still green (316 download tests).	4 weeks ago
Broque Thomas	73fb60a68a	C3: Migrate Tidal to engine.worker Same pattern as C2 — TidalDownloadClient drops active_downloads + _download_lock + _download_thread_worker. download() delegates to engine.worker.dispatch with _download_sync as the impl. Source-specific extras (track_id, display_name) merge into the engine record. The HLS-segment progress callback (_update_download_progress) now writes to engine state via engine.update_record instead of mutating the per-client dict in-place. Query/cancel methods (get_all_downloads, get_download_status, cancel_download, clear_all_completed_downloads) now read engine state via the same accessors as the YouTube migration. Pinning tests updated to assert engine state. Suite still green (313 download tests). Behavior preserved end-to-end.	4 weeks ago
Broque Thomas	4ddfb01a0a	C2: Migrate YouTube to engine.worker YouTubeClient drops its hand-rolled background thread + state dict + semaphore + last-download-timestamp. download() now delegates to engine.worker.dispatch with _download_sync as the impl callable; YouTube-specific record fields (video_id, url, title) merge into the engine record via extra_record_fields. Engine wires itself in via plugin.set_engine(engine) callback on register_plugin. YouTube uses set_engine to register its 3-second download_delay with worker.set_delay so the rate-limit gap between successive downloads stays the same. Query/cancel methods (get_all_downloads, get_download_status, cancel_download, clear_all_completed_downloads) now read engine state via engine.iter_records_for_source / get_record / update_record / remove_record. Net: ~120 LOC of thread+state boilerplate removed from youtube_client.py. Phase A pinning tests updated to assert engine state instead of client.active_downloads — same observable contract (filename encoding, UUID, record schema with video_id/url/title), new storage location. Suite still green (2025 passed). Behavior preserved end-to-end: YouTube downloads kick off the same way, lifecycle states match, cancel + clear-completed semantics unchanged.	4 weeks ago
Broque Thomas	78724861f9	C1: Add BackgroundDownloadWorker to engine `BackgroundDownloadWorker` lives on the engine and owns the boilerplate every streaming download client currently hand-rolls: thread spawn, per-source semaphore, rate-limit delay, state lifecycle (Initializing → InProgress → Completed or Errored), exception capture. Plugins provide only the atomic download op (`impl_callable`). Per-source rate-limit policy (concurrency, delay) is configured on the worker via `set_concurrency` / `set_delay`. Source- specific record fields merge in via `extra_record_fields` so existing consumer code that reads `video_id`, `track_id`, `permalink_url`, etc. keeps working post-migration. Username slot supports override (Deezer's legacy `'deezer_dl'`). Phase C1 scope: worker exists. No client migrated yet — C2-C7 migrate sources one at a time, each gated by the Phase A pinning tests so per-source contract drift fails fast. 10 new tests pin the worker contract: UUID id format, initial record shape, extra-fields merge, username override, state transitions on success / impl-returns-None / impl-raises, semaphore serialization (default + parallel), rate-limit delay between successive downloads. Suite still green (308 download tests). Pure additive.	4 weeks ago
Broque Thomas	3634dca83f	B3: Orchestrator delegates query methods to engine `get_all_downloads`, `get_download_status`, `cancel_download`, and `clear_all_completed_downloads` on the orchestrator are now thin pass-throughs to the engine. The plugin-iteration logic lives in one place (the engine) instead of duplicated across orchestrator methods. Source-hint routing semantics preserved verbatim — engine.cancel treats streaming-source names as direct routes and unknown names as Soulseek peer usernames, exactly like the legacy orchestrator did. Per-plugin exceptions still get swallowed defensively. Test fixture `_build_orchestrator` now constructs an engine and registers every mock plugin so the helper-built orchestrators have the same wiring as production. Suite still green (2012 passed). Zero behavior change for users.	4 weeks ago
Broque Thomas	badb5dd7de	B2: Engine owns cross-source query dispatch `DownloadEngine` grows async query methods that wrap plugin iteration: `get_all_downloads` (concatenates every plugin's active downloads), `get_download_status` (first plugin to recognize the id wins), `cancel_download` (with source-hint routing — streaming sources go direct, unknown hints route to Soulseek as peer username), and `clear_all_completed_downloads` (skips unconfigured plugins). Code moved from the orchestrator's hand-iterated loops into the engine. Orchestrator delegation comes in B3 — for B2 the engine methods exist but nothing calls them yet. Per-plugin behavior preserved verbatim (defensive `try ... except` swallows per-iteration, unconfigured-skip on clear, source-hint routing semantics). Phase A pinning tests + 8 new engine query tests catch any drift. Pure additive — zero behavior change for users.	4 weeks ago
Broque Thomas	f40c6d3b55	B1: Add DownloadEngine skeleton `core/download_engine/` package with the engine class that will own cross-source state, threading, search retry, rate-limits, and fallback chains. Orchestrator constructs an engine and registers each plugin with it. Phase B1 scope: skeleton only. Engine stores active_downloads records keyed by (source, download_id), provides thread-safe add/update/remove/iterate primitives, and holds plugin references for later phases. NOT on any code path yet — pure additive scaffolding so subsequent commits can introduce engine-driven behavior one piece at a time without a big-bang switchover. 15 new tests pin the engine's state-storage contract: shallow-copy reads, partial-patch updates, no-op-on-missing semantics, per-source iteration, id-only find, concurrent-add safety. Suite still 290 (download subset) green. Zero behavior change.	4 weeks ago
Broque Thomas	4c2fd49df2	A8: Pin LidarrDownloadClient download lifecycle behavior 6 tests pin the Lidarr contract — the special case in the dispatcher because Lidarr is an ALBUM-grabber not a track-grabber. Filename format is `album_foreign_id\|\|display` (MusicBrainz album MBID Lidarr uses for lookups). State dict is SMALLER than streaming sources (no track_id, no transferred/speed — Lidarr polls its own queue API for byte-level progress). Thread target signature is 3-arg, no original_filename. Engine refactor's plugin contract must accommodate album-only sources or Lidarr stays special.	4 weeks ago
Broque Thomas	2a0d63723e	A7: Pin SoundcloudClient download lifecycle behavior 6 tests pin the SoundCloud contract: 3-part filename `track_id\|\|permalink_url\|\|display_name` (yt-dlp consumes the URL, not the track_id). Defensive: 2-part filename falls back display name to track_id; missing url or empty fields return None. Thread target signature uses URL as the second arg.	4 weeks ago
Broque Thomas	07834ff4f0	A6: Pin DeezerDownloadClient download lifecycle behavior 6 tests pin the Deezer contract: - track_id stays as STRING (Deezer GW API uses string IDs). - username slot is the legacy `'deezer_dl'` (frontend depends on it). - Auth gate at top of `download()` returns None BEFORE thread spawn. - Defensive fallback: filename without `\|\|` synthesizes display name. - Thread is named `deezer-dl-<track_id>` for diagnostics. - State dict has Deezer-specific `error` slot.	4 weeks ago
Broque Thomas	6667c079ae	A5: Pin HiFiClient download lifecycle behavior 5 tests pin the HiFi contract: int track_id, UUID download_id, state-dict schema, daemon-thread worker. Note: target method is `_download_worker` (NOT `_thread_worker` like Tidal/Qobuz) and worker signature is 3-arg (download_id, track_id, display_name). Engine refactor's plugin contract must accommodate or normalize.	4 weeks ago
Broque Thomas	be81bf05d4	A4: Pin QobuzClient download lifecycle behavior 5 tests pin the Qobuz contract: int track_id parsing, UUID download_id, state-dict schema (parallels Tidal), daemon-thread worker target with (download_id, track_id, display_name, original_filename) signature.	4 weeks ago
Broque Thomas	366ee445c7	A3: Pin TidalDownloadClient download lifecycle behavior 8 tests pin the Tidal contract: filename encoding (`<int>\|\|display` where track_id parses as int), UUID download_id format, initial state-dict schema, daemon-thread spawn semantics, and the active_downloads → DownloadStatus translation. is_authenticated false on no-session AND on tidalapi.check_login() exceptions (orchestrator skip behavior depends on this).	4 weeks ago
Broque Thomas	5e6d0bdf0d	A2: Pin YouTubeClient download lifecycle behavior 5 tests pin the YouTube download contract: filename encoding (`video_id\|\|title`), UUID download_id format, initial state-dict schema, daemon-thread spawn for background work, and the `_download_thread_worker` target shape. Phase C will replace the thread spawn with `engine.dispatch_download` — these tests catch any drift in the per-download record shape that consumers depend on. Pure additive — no client code changes.	4 weeks ago
Broque Thomas	52ab9aeb5b	A1: Pin SoulseekClient download lifecycle behavior 13 tests pin slskd HTTP API contract: endpoint format (`transfers/downloads/<username>` POST), payload shape (slskd web-interface array format), id extraction from dict / list / fallback responses, and the username-lookup fallback in cancel_download when no username hint is provided. Phase A of the download engine refactor — pinning current behavior of every source BEFORE moving any code so the engine extraction can't drift the per-source contract. Includes the plan doc at docs/download-engine-refactor-plan.md. Pure additive — no client code changes.	4 weeks ago
Broque Thomas	f9b763587d	Add plugin conformance tests + WHATS_NEW entry 19 parametrized tests pin every registered plugin class's structural conformance to DownloadSourcePlugin: every required method present + async-ness matches the protocol. Drift in any source fails at the test boundary instead of at runtime against a live download. Class-level checks (not instance-level) — instantiating real clients in fixtures pollutes module state via tidalapi etc. imports and breaks downstream tests.	4 weeks ago
Broque Thomas	5294065fe4	Wire orchestrator through plugin registry Every per-source dispatch site (search, download, get_all_downloads, get_download_status, cancel_download, clear_all_completed_downloads, cancel_all_downloads, reload_settings) now iterates `registry.all_plugins()` instead of hand-maintained client lists. Backward-compat `self.soulseek` / `self.youtube` / etc. attributes preserved as registry-resolved aliases — external callers reaching for source-specific internals (e.g. `orchestrator.soulseek._make_request`) keep working unchanged. Adding a new source (Usenet planned) becomes one registry entry + the new client class — no orchestrator changes.	4 weeks ago
Broque Thomas	05bfb724a8	Update mbid consistency test mock to match new create_finding bool contract	4 weeks ago
Broque Thomas	cf5461f2f1	Fix: maintenance findings badge inflated when scan dedup-skipped `_create_finding` silently dedup-skipped re-discovered issues but the caller incremented `findings_created` regardless. So a re-scan that found the same issues as a prior scan reported 364 findings in the badge while 0 NEW pending rows hit the db, leaving the findings tab empty. `_create_finding` now returns bool (True on insert, False on dedup-skip / db error). All 16 repair jobs updated to only increment `findings_created` on True. Added `findings_skipped_dedup` counter surfaced in scan log: "Done: X scanned, 0 fixed, 0 findings (363 already existed), 0 errors". Also fixed a missing `job_id` kwarg in album_tag_consistency that was silently breaking finding creation for that scan.	4 weeks ago
Broque Thomas	77c54ab7a7	Migrate discography + quality scanner to typed Album path Three more album-shape consumers now route through Album.from_<source>_dict() when caller passes a known source: - _build_discography_release_dict (artist discography cards) - _build_artist_detail_release_card (artist detail release cards) - _normalize_track_album (quality scanner result normalization) Legacy duck-typing stays as fallback for unknown source, non-dict input, or converter errors. Pure additive — existing callers without source kwarg unchanged.	4 weeks ago
Broque Thomas	967c7f7c0a	Migrate album-info builders to typed Album path Steps 2+3 of typed metadata migration. Two album-info builders now route through Album.from_<source>_dict() when caller passes a known source: - _build_album_info (album-tracks lookups) - _build_single_import_context_payload (single-track import context) Legacy duck-typing stays as fallback for unknown source, non-dict input, or converter errors. Pure additive — existing callers without source kwarg unchanged.	4 weeks ago
Broque Thomas	eab1297afc	Add Qobuz + Tidal album converters Audit caught two missing providers from the foundation pr. Both return album-shaped data via their clients (search + download flows). Tidal uses tidalapi objects rather than dicts so the converter is from_tidal_object, not _dict. Enrichment-only providers (lastfm/genius/acoustid/listenbrainz/ audiodb) intentionally have no album converter — they enrich existing rows, never return album shapes. Tests: +8 cases. 40 total now.	4 weeks ago
Broque Thomas	529486a2d1	Foundation: typed Album/Track/Artist + per-provider converters New core/metadata/types.py with canonical dataclasses + classmethod converters for spotify/itunes/deezer/discogs/musicbrainz/hydrabase. Each converter is the single place that knows that provider's wire shape — addresses the duck-typing pattern Cin flagged. Pure additive: no consumer code changed. Follow-up PRs migrate consumers one at a time. Migration plan at docs/metadata-types-migration.md. Tests: 32 cases pin per-provider semantics + cross-provider invariants. Also stabilized a flaky discogs test that depended on local config state.	4 weeks ago
Broque Thomas	4b23bee4a9	Add Discogs collection as a Your Albums source Discord request: pull user's Discogs collection into the Your Albums section on Discover, similar to how Spotify Liked Albums works. Implementation extends the existing 3-source pipeline (Spotify / Tidal / Deezer) to a 4-source pipeline with click-context dispatch — Discogs-only albums open with rich Discogs release detail (vinyl/CD format, year, label, country, tracklist). Mirrors the per-source dispatch pattern from enhanced/global search. Discogs client (`core/discogs_client.py`): - New `get_authenticated_username()` resolves the username for the configured personal token via Discogs's `/oauth/identity` endpoint. Cached on the instance so subsequent collection page-fetches don't re-hit it. - New `get_user_collection(username=None, folder_id=0, per_page=100, max_pages=50)` walks all pages of `/users/{username}/collection/ folders/{folder_id}/releases`. Returns normalized dicts ready for upsert_liked_album. folder_id=0 = Discogs's "All" folder. Pagination cap of max_pages*per_page = 5000 releases — bounds runtime on heavy collections. - New `get_release(release_id)` thin wrapper for `/releases/{id}` — returns the raw API response so the album-detail endpoint can render rich context. - Both methods defensive: missing token → empty list, malformed responses → skipped, falsy ids → None. Disambiguation suffix stripping (`Madonna (3)` → `Madonna`) so Discogs artist names match what Spotify/Tidal/Deezer use. Schema (`database/music_database.py`): - New `discogs_release_id TEXT` column on `liked_albums_pool`. Migration uses the established `try SELECT, except ALTER TABLE` pattern. Idempotent; safe on existing installs. - Added the column to the canonical CREATE TABLE for fresh installs. - `upsert_liked_album` extended with `'discogs': 'discogs_release_id'` in BOTH the INSERT and UPDATE id-column maps so Discogs source_id routes to the new column. INSERT statement column count + value count updated together. Backend (`web_server.py`): - `/api/discover/your-albums/sources` — adds Discogs to the `connected` list when `discogs.token` config is set. - `_fetch_liked_albums` — new branch for Discogs. Lazy-imports DiscogsClient, respects the `enabled_sources` config, walks the collection, upserts each release. Same try/except shape as the existing source branches. - `/api/discover/album/<source>/<album_id>` — new `discogs` branch fetches the release via DiscogsClient.get_release, normalizes the Discogs tracklist format, parses Discogs's `MM:SS`/`HH:MM:SS` duration strings to milliseconds, returns the same response shape as the Spotify/Deezer/iTunes branches. Frontend (`webui/static/discover.js`): - `openYourAlbumsSourcesModal` — adds Discogs to `sourceInfo` with the vinyl emoji icon. Existing toggle/save plumbing handles it. - `openYourAlbumDownload` — restructured the per-source dispatch: builds an ordered list of (source, id) tuples, tries each in turn, breaks on the first successful response. Pure-Discogs albums go straight to the Discogs detail endpoint → modal opens with Discogs context. Multi-source albums prefer Spotify/Deezer first since their tracklists carry proper streaming IDs ready for download. Tests: `tests/test_discogs_collection_source.py` — 12 cases: - get_user_collection: empty without token, normalizes response shape, strips disambiguation suffix, handles missing year, skips malformed releases, paginates correctly, caps at max_pages, uses explicit username when provided. - get_release: passes id through to /releases/{id}, returns None for invalid ids without API call. - liked_albums_pool: discogs_release_id round-trips through upsert + get; multi-source dedup carries both Spotify and Discogs IDs on the same row. Verified: full suite 1825 pass (12 new), ruff clean, smoke test populating + reading the discogs_release_id column round-trips correctly via the real DB. WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	2ab460f5c4	Add Library Disk Usage card to System Statistics Discord request (Samuel [KC]): show how much disk space the library takes on the Stats page. Implementation piggybacks on the existing deep scan — Plex/Jellyfin/Navidrome all return file size in their track API responses, so we read it during the deep scan and store it on the tracks row. Aggregation is then a single SQL query — no filesystem walk, no extra I/O during the scan, no separate stat job. SoulSync standalone gets size from os.path.getsize at insert time (different code path; the file is local when we write the row). Schema (`database/music_database.py`): - New `file_size INTEGER` column on `tracks`. Migration uses the established `try SELECT, except ALTER TABLE ADD COLUMN` pattern. Idempotent; safe on existing installs. NULL on legacy rows so they don't contribute to totals until next deep scan refreshes. - Added the column to the canonical CREATE TABLE so fresh installs get it without going through the migration path. Track-object plumbing: - `core/jellyfin_client.py` — JellyfinTrack reads MediaSources[0].Size alongside existing Bitrate read. None when 0 / missing. - `core/navidrome_client.py` — NavidromeTrack reads `size` from the Subsonic song object (int coercion + None on parse fail). - `core/soulsync_client.py` — SoulSyncTrack does os.path.getsize (only "server" where size has to come from disk). - Plex needs no client-side change: track.media[0].parts[0].size is read directly inside insert_or_update_media_track. Persistence — TWO separate insert paths: (a) `database/music_database.py:insert_or_update_media_track` — Plex/Jellyfin/Navidrome flows. Reads file_size from Plex's MediaPart OR `track_obj.file_size` wrapper attribute (defensive Plex-attr-not-present check + > 0 type guard). INSERT writes the new column. UPDATE uses COALESCE(?, file_size) so a None from the server on a re-sync (rare Jellyfin Size omission) doesn't blank an existing value. Pinned via test. (b) `core/imports/side_effects.py:record_soulsync_library_entry` — SoulSync standalone flow. Completely separate code path: the standalone deep scan moves files to staging for auto-import rather than calling insert_or_update_media_track. After the auto-import processes them, side_effects writes the tracks row directly. Reads file_size via os.path.getsize(final_path) at insert time (file is local) and includes it in the INSERT column list. SoulSync only does INSERT-if-not-exists (no UPDATE path), so no COALESCE concern. Aggregator (`database/music_database.py:get_library_disk_usage`): - SELECT COALESCE(SUM(file_size), 0), COUNT(file_size), COUNT() - COUNT(file_size) for the totals. - Per-format breakdown done in Python via os.path.splitext over (file_path, file_size) rows — sidesteps SQLite's first-vs-last-dot ambiguity for paths like /music/Kendrick/M.A.A.D City/01.flac. - Defensive: skips empty paths, paths without extension, and implausibly long extensions (>6 chars). Returns the full empty-shape dict (NOT a partial / undefined) when the column doesn't exist or queries fail, so the UI's `if (!data.has_data)` branch handles fresh installs cleanly. API + UI: - `core/stats/queries.py` — thin pass-through get_library_disk_usage matching the existing query-helper convention. - `web_server.py` — new /api/stats/library-disk-usage endpoint mirroring the /api/stats/db-storage pattern. - `webui/index.html` — new card in System Statistics above the Database Storage card. - `webui/static/stats-automations.js` — _loadLibraryDiskUsage + _renderLibraryDiskUsage. Empty state: "Run a Deep Scan to populate (X tracks pending)". Partial: "X measured (+Y pending)". Full: total + format bars proportional to the largest format. - `webui/static/style.css` — .stats-disk- styled to match the Database Storage card. Backward compatibility: - Migration is additive; existing rows get NULL file_size; the empty-shape return from the aggregator means the UI renders cleanly without errors before any deep scan runs. - Old installs upgrading will see "Run a Deep Scan to populate (N tracks pending)". Running their next deep scan fills sizes — the existing scan flow doesn't need any changes, just consumes the new track-wrapper attribute. Tests: - `tests/test_library_disk_usage.py` — 13 cases covering schema migration, NULL defaults on legacy inserts, fresh-install empty shape, summing with mixed NULL/known sizes, per-format breakdown, mixed-case extensions, paths with album-name dots, missing extensions, empty file_path, implausibly long extensions, JellyfinTrack.file_size persistence via insert_or_update_media_track, COALESCE preservation on null re-sync. - `tests/imports/test_import_side_effects.py` — extended the existing record_soulsync_library_entry test to assert track_row['file_size'] == os.path.getsize(final_path), pinning the SoulSync-standalone path. Test fixture's tracks schema also updated to include the file_size column. Verified: full suite 1813 pass (13 new, 1 existing-test extension), ruff clean, smoke test populating + reading the column round-trips correctly. WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	776d195f71	Fix: ReplayGain wrote same +52 dB gain to every track User report: every downloaded track in an album came out with ``replaygain_track_gain: +52.00 dB`` regardless of actual loudness. Root cause: the parser at ``core/replaygain.py:79`` used ``re.search('I:\s+...')`` which returns the FIRST match. ffmpeg's ebur128 filter emits ``I:`` per measurement window (running partial integrated loudness) AND in a final Summary block. The first per-window reading is at t=0.5s — almost always ~-70 LUFS because nearly every track starts with silence / encoder padding. So: gain = RG2_reference - lufs = -18 - (-70) = +52.00 dB …on EVERY track. Same regex pattern, same first per-window match, same +52 dB written to every file's REPLAYGAIN_TRACK_GAIN tag. Verified by running ffmpeg ebur128 against a real generated FLAC and inspecting the stderr output — first per-window line at t=0.5s shows ``I: -70.0 LUFS`` (silent intro), and the Summary block at the end shows the real integrated value (e.g. ``I: -27.8 LUFS`` for the test sine wave). Old code captured the -70.0 reading. Fix: anchor LUFS parsing to the ``Summary:`` block via ``stderr.rfind('Summary:')``. The Summary block is always emitted last and contains the authoritative final integrated loudness. Peak parsing already worked correctly (per-window output uses ``TPK:``/``FTPK:`` labels; only the Summary uses ``Peak:``), but applied the same Summary anchor for consistency. Defensive fallback: if no Summary block is present (truncated output / unusual ffmpeg version), use the LAST per-window reading instead of the first. Still better than the buggy first-window behavior. Smoke verified end-to-end: a freshly-generated FLAC of a -24 dBFS sine wave now reports LUFS=-27.80, gain=+9.80 dB (correct, was +52.00 before fix). Tests: ``tests/test_replaygain_summary_parse.py`` — 7 cases pinning the parser behavior with realistic ffmpeg ebur128 stderr samples: - Summary value parsed correctly even when first per-window is -70 - Resulting gain is realistic (NOT +52) - Two tracks with same first per-window but different summaries get different LUFS (regression assertion for "all tracks same gain") - Per-window reading higher than Summary doesn't leak through - Fallback to last per-window when Summary absent - Clean RuntimeError raised when no LUFS values anywhere - Peak still correctly anchored to Summary Verified: full suite 1800 pass (7 new), ruff clean. WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	04a14f7e96	Fix: tasks showed Completed when file was quarantined User caught downloading Kendrick Mr. Morale: three tracks (Rich Interlude, Savior Interlude, Savior) showed ✅ Completed in the modal but were missing on disk. Log forensics revealed two layered bugs. Bug 1 — Verification wrapper assumed success on quarantined files (`core/imports/pipeline.py`): The outer `post_process_matched_download_with_verification` had a fallback at the "no `_final_processed_path` in context" branch that marked the task completed and notified `success=True`. The inner post-processor sets `_final_processed_path` only when the file actually reaches its destination. Integrity-rejected files (`_integrity_failure_msg` set) and race-guard-failed files (`_race_guard_failed` set) get quarantined or skipped without ever setting `_final_processed_path`, so they fell straight into the "assume success" branch. Confirmed in user's log: No _final_processed_path in context for task d5b88b84-... — cannot verify, assuming success That line fired for the same task right after the integrity check quarantined the source file. Result: ✅ Completed in UI, file in quarantine, never delivered. Fix: explicit checks for `_integrity_failure_msg` and `_race_guard_failed` markers BEFORE the assume-success fallback. Either marker set → task status='failed' with descriptive error_message + `_notify_download_completed(success=False)`. The pre-existing assume-success behavior preserved when no failure markers are set (some legitimate flows complete without setting `_final_processed_path`). Bug 2 — AcoustID skip-logic too lenient (`core/acoustid_verification.py`): The "language/script" exemption was: if best_score >= 0.95 and (title_sim >= 0.55 or artist_sim >= ARTIST_MATCH_THRESHOLD): The OR-clause fired for English-vs-English titles by the same artist that share NO actual content. Confirmed in user's log: requested "Rich (Interlude)" by Kendrick Lamar, AcoustID identified the audio as "R.O.T.C. (interlude)" by Kendrick Lamar (a totally different song from his 2010 mixtape) — same artist scored ≥ARTIST threshold, shared word "interlude" pushed title_sim above 0.55, skip fired. Verification returned SKIP instead of FAIL, the wrong file was accepted as the answer for three different track requests. Fix: skip now requires positive evidence the mismatch is a real language/script case: (a) Non-ASCII chars present in either title AND artist matches strongly → real transliteration case (kanji ↔ romaji etc) (b) BOTH title_sim >= 0.80 AND artist_sim >= ARTIST threshold → minor punctuation/casing differences English-vs-English with very different titles by the same artist no longer skipped — verification correctly returns FAIL, the wrong file gets quarantined, the new wrapper logic above marks the task failed. Tests: - `tests/test_integrity_failure_marks_task_failed.py` — 4 cases pinning the wrapper-level state machine: integrity marker → failed, race-guard marker → failed, no markers → still assumes success (legacy path preserved), integrity-failure-takes-priority over missing-final-path fallback. - `tests/test_acoustid_skip_logic.py` — 7 cases pinning the skip exemption: user's R.O.T.C-vs-Rich case → FAIL (regression test), Savior-vs-R.O.T.C → FAIL (same bug surface), Japanese kanji → romaji → SKIP (real language case still works), MAAD vs M.A.A.D → PASS or SKIP (punctuation tolerance), low fingerprint score → never skipped, high score but artist mismatch → no longer skipped, Crown vs Crown of Thorns → no longer skipped. Verified: full suite 1793 pass (11 new), ruff clean. WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	4b15fe0b75	Fix album MBID inconsistency: detector + persistent release-MBID cache Discord report (Samuel [KC]): tracks of the same album sometimes carry different MUSICBRAINZ_ALBUMID tags, which causes Navidrome (and other media servers grouping by album MBID) to split the album into multiple entries. Two-part fix — one for existing libraries, one for the root cause that lets new imports drift. Part 1 — Detector + fix action (catches existing dissenters): `core/repair_jobs/mbid_mismatch_detector.py`: - New helpers: `_read_album_mbid_from_file` and `_write_album_mbid_to_file` use the Picard-standard tag conventions (`TXXX:MusicBrainz Album Id` for MP3, `MUSICBRAINZ_ALBUMID` for FLAC/OGG, `----:com.apple.iTunes:MusicBrainz Album Id` for MP4). - New scan phase `_scan_album_mbid_consistency` runs after the existing track-MBID scan: groups tracks by DB `album_id`, reads each track's embedded album MBID, finds the consensus (most-common) MBID via `Counter`, flags dissenters. Tracks without an album MBID at all are skipped (they don't break Navidrome — only an explicit MBID disagreement does). Albums where MBIDs are perfectly tied (no clear consensus) are skipped too — surface as a manual decision instead of fixing toward a 1/N tie. - New finding type `album_mbid_mismatch` carries `consensus_mbid`, `wrong_mbid`, `consensus_count`, `total_tracks_with_mbid`, and a human-readable reason string. `core/repair_worker.py`: - Added `'album_mbid_mismatch': self._fix_album_mbid_mismatch` to the fix dispatch dict and to the `fixable_types` tuple so auto-fix + bulk-fix paths pick it up. - New `_fix_album_mbid_mismatch` method reads `consensus_mbid` from finding details, resolves the dissenter's file path via the shared library resolver, calls `_write_album_mbid_to_file` to rewrite the tag in place. Doesn't touch the album's other tracks (they're already in agreement). Part 2 — Root cause fix (prevents new SoulSync imports from drifting): The original in-memory `mb_release_cache` in `core/metadata/source.py` maps `(normalized_album, artist) -> release_mbid` so per-track enrichment of the same album hits the cache and writes the same MUSICBRAINZ_ALBUMID to every track. That cache is bounded (4096 entries) and in-process — so cache eviction (when other albums are processed in between) and server restart can BOTH cause inconsistency. Per-track album-name variation (e.g. some tracks tagged `"Album"`, others tagged `"Album (Deluxe)"`) and per-track artist variation (features) make it worse. `core/metadata/album_mbid_cache.py` (new module): - DB-backed `lookup(normalized_album, artist) -> release_mbid` and `record(...)` functions. Same key shape as the in-memory cache. - Strict additive design: every public function is wrapped in try/except and degrades to None / no-op on ANY database error. The existing in-memory cache + MusicBrainz lookup remains the authoritative fallback. If this module breaks, downloads continue exactly as they would today. `database/music_database.py`: - New `mb_album_release_cache` table with composite primary key `(normalized_album_key, artist_key)`. Reverse-lookup index on `release_mbid` for future debug tooling. Created via the existing `CREATE TABLE IF NOT EXISTS` migration pattern — idempotent, no schema version bump needed. `core/metadata/source.py`: - Surgical change inside the existing `embed_source_ids` in-memory-cache-miss branch: BEFORE calling MusicBrainz, consult the persistent cache. If a previous SoulSync run already resolved this album's release MBID, reuse it. After a successful MB lookup, store in BOTH caches. Both calls wrapped in defensive try/except so any failure falls through to existing logic. Tests: - `tests/metadata/test_album_mbid_cache.py` — 16 cache tests: round-trip, idempotent re-record, overwrite semantics, clear_all, album+artist independence (no Greatest Hits collisions), defensive None-on-empty-input, graceful degradation when the DB is unavailable / connection raises / commit fails, schema sanity (table + index exist after init). - `tests/test_album_mbid_consistency.py` — 13 detector tests: tag read/write round-trip on real FLAC files, Picard-standard tag descriptors, defensive paths (unreadable file, empty input), detector behavior (agreement → no flags, lone dissenter → flag, ties → no flag, single-track albums → skipped, no-MBID tracks → skipped, unresolvable file paths → skipped). - `tests/metadata/test_metadata_enrichment.py` — added autouse fixture monkeypatching the persistent cache to no-op for tests in this file. The existing tests pin per-call MB counts and in-memory cache state; without the fixture, persistent rows from earlier tests would bypass the MB call. Persistent layer has its own dedicated tests. Verified: 1782 tests pass (29 new), ruff clean, smoke test confirms end-to-end cache round-trip works. WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	e577f3cf1f	Fix three Lidarr bugs that prevented it from being a real download source Investigation surfaced that Lidarr was wired into the orchestrator but the actual download flow had blockers: 1. Wrong file misfiled. Lidarr grabs whole albums; SoulSync's matched-context post-processing wants the SPECIFIC track the user requested. Old code copied every track in the album and reported `imported_files[0]` as `file_path` — almost always pointing to track 1, not the user's actual track. Post-processing then tagged track 1 with the requested track's metadata. Misfiling on every real download. Fix: parse the wanted track title out of the dispatch display name (which `_search_sync` already builds as `f"{artist} - {album} - {track_title}"`), look it up against Lidarr's `track` API, resolve the matching `trackFileId` to a path, and copy ONLY that file. Punctuation-tolerant fuzzy match handles the common "m.A.A.d city" vs "maad city" case. Album-level dispatches (no track in the display) preserve the old first-file fallback so existing album-grab UX is unchanged. 2. Hardcoded `metadataProfileId=1`. Required by Lidarr's artist-add API. On installs where the user deleted/recreated metadata profiles, that id no longer exists and the call fails with HTTP 400 — which silently breaks every download flow that needs to add an artist. Real-world Lidarr installs do this all the time. Fix: `_get_metadata_profile_id()` calls Lidarr's `metadataprofile` API and returns the first available id. Falls back to 1 only when the API call fails entirely (preserves previous behavior so this change can't make things worse). 3. Polling never broke the outer loop on completion. The inner `for item in queue['records']` had `break` statements at status transitions, but those only escaped the queue iteration — the outer `for poll in range(max_polls)` kept spinning until the 600-poll timeout even after the album was clearly imported. `for/else` semantics didn't apply because completion was detected inside the inner loop, not by it running to exhaustion. Fix: replaced with an explicit `download_complete` flag set when `album/{id}` reports `trackFileCount > 0` (the authoritative completion signal — works even when the queue record disappeared between polls). Outer loop breaks immediately once the flag flips. Helper functions added: `_extract_wanted_track_title` (staticmethod, splits the display name; >=3 parts → track dispatch, 2 parts → album dispatch), `_normalize_for_match` (lowercase + strip punctuation + collapse whitespace for fuzzy compare), `_title_similarity` (cheap score: equal=1.0, substring=0.85, token-overlap-ratio otherwise), `_pick_track_file_for_wanted` (orchestrates the API calls). Settings tooltip updated to be honest about Lidarr's natural shape: album-grabber, no-op for playlist sync, hybrid mode falls through to other sources for track searches. Sets correct expectations. Tests: `tests/test_lidarr_download_client.py` — 21 isolated tests covering pure helpers (title extraction, normalization, similarity) and the file-picker integration paths (matching path, punctuation tolerance, below-threshold fallback, missing trackFileId, missing file on disk, API failures, malformed responses). No live Lidarr needed — `_api_get` mocked at the client boundary. Isolation: ONLY touches `core/lidarr_download_client.py`, the Lidarr settings tooltip in `webui/index.html`, the Lidarr WHATS_NEW entry in `webui/static/helper.js`, and the new test file. No changes to the orchestrator, other download clients, the import pipeline, side_effects, web_server.py, settings.js, or any shared validation / monitor / task_worker code. Other download sources are not affected in any way. Verified: 1753 tests pass (21 new), ruff clean.	4 weeks ago
Broque Thomas	8de4a186b7	Fix three SoundCloud integration gaps surfaced by smoke testing User report: switched download source to SoundCloud and noticed: 1. Download progress % stays at 0 until "suddenly done" — no live progress 2. Sidebar status indicator next to "SoundCloud" label is red 3. Dashboard service status card still shows "Soulseek" as the source name Fix 1 — Live progress for HLS-segmented SoundCloud downloads (`core/soundcloud_client.py`): - yt-dlp's `total_bytes` / `total_bytes_estimate` for HLS describes the CURRENT FRAGMENT, not the whole download. So the byte-based percentage stayed near 0 the entire time — until 'finished' fired. - Added `_update_download_progress_fragmented` which uses `fragment_index` / `fragment_count` (which yt-dlp DOES populate accurately for HLS) to compute a meaningful percentage. Total size is extrapolated from per-fragment average for the bytes/remaining display. Time-remaining estimate uses elapsed/index seconds-per- fragment. - The progress hook prefers fragment progress when both fragment_index and fragment_count are present; falls back to byte-based for non-fragmented (progressive MP3) downloads. Five new unit tests pin the fragment-progress math, the 99.9% cap, and the defensive zero-index / unknown-id paths. Fix 2 — Sidebar status indicator stays green for SoundCloud mode (`web_server.py`): - The `/api/status` route's `serverless_sources` tuple decides whether to even probe slskd. SoundCloud (and Lidarr) were missing — so when the active source was SoundCloud, the route fell through to "test slskd, mark not-relevant", which set `connected: False` and turned the sidebar dot red even though SoundCloud was working. - Added `'soundcloud'` and `'lidarr'` to the tuple. Both are serverless from slskd's perspective, so the dot now stays green whenever they're the active source. Fix 3 — Dashboard service card title shows the active source (`webui/static/shared-helpers.js`): - The dashboard's "Download Source" card has its own `sourceNames` map at line 3351 (separate from the sidebar map I already updated at 3396). Missed it during the integration PR. - Added `'lidarr'` and `'soundcloud'` so the card title now reads "SoundCloud" / "Lidarr" instead of falling back to "Soulseek". Bonus — Dashboard "Test Connection" button works for SoundCloud (`core/connection_test.py`): - The dashboard's Test Connection button on the download-source card sends `service` based on the active source — so for SoundCloud it was sending `service='soundcloud'`. `run_service_test` had no branch for it, so it fell through to "Unknown service." and the button always failed. - Added a `soundcloud` branch that mirrors `/api/soundcloud/status` behavior: confirms yt-dlp is installed, runs a real cheap probe, returns a meaningful pass/fail. (HiFi has the same gap but no user reported it; out of scope for this fix.) Verified: - 41 unit tests pass (5 new fragment-progress tests added) - Full suite 1732 passed - Ruff clean	4 weeks ago
Broque Thomas	75fe04907f	Wire SoundCloud as a first-class download source Plug the previously-built SoundcloudClient (PR #478, the build-and-verify phase) into every place a download source needs to appear. Follows the same wiring contract as Tidal/Qobuz/HiFi/Deezer/Lidarr — orchestrator routing, hybrid-mode picker, search dispatch, queue/cancel/clear, provenance + library history, sidebar source label, settings UI all work plug-and-play. Backend wiring: - `core/download_orchestrator.py` — import SoundcloudClient, _safe_init it at startup, add to _client() lookup, get_source_status(), check_connection's sources_to_check default, search source_names map, search_and_download_best _streaming_sources tuple, download source_map + source_names, and every iteration loop in reload_settings download-path-update / get_all_downloads / get_download_status / cancel_download (route + iterate) / clear_all_completed_downloads / cancel_all_downloads. - `core/downloads/monitor.py` — added SoundCloud to the per-client loop that fetches active downloads outside the orchestrator (uses getattr fallback for older soulseek_client snapshots). - `core/downloads/task_worker.py` — added SoundCloud (and Lidarr, which was missing too — bonus fix) to source_clients dict for hybrid fallback dispatch. - `core/downloads/validation.py` — added 'soundcloud' to _streaming_sources so SoundCloud results go through the matching engine validation path instead of the Soulseek quality-filter path. - `core/imports/side_effects.py` — three call sites: source_map for download_source label written to library_history, streaming-source guard for the `\|\|`-encoded stream_id parsing, and source_service map for provenance recording. All three now include 'soundcloud'. - `web_server.py` — five streaming-source detection tuples updated. New `/api/soundcloud/status` endpoint returns {available, configured, reachable} mirroring the Deezer/HiFi status-endpoint pattern; reachability runs a real cheap yt-dlp search so the settings Test Connection button gives a meaningful pass/fail signal. - `config/settings.py` — added empty `soundcloud_download` defaults block so future tier-2 OAuth (SoundCloud Go+ session) doesn't have to migrate existing configs. Frontend: - `webui/index.html` — new `<option value="soundcloud">` in the download-source-mode dropdown, SoundCloud added to both hidden legacy hybrid-source selects, new settings container with info text + Test Connection button. - `webui/static/settings.js` — HYBRID_SOURCES entry (with the SoundCloud cloud SVG icon), _hybridSourceEnabled default, updateDownloadSourceUI container display, allSources for legacy hybrid picker, testSoundcloudConnection function (hits the new status endpoint, color-codes the result), saveSettings soundcloud_download empty block. - `webui/static/shared-helpers.js` — sidebar source-name map includes SoundCloud + Lidarr (Lidarr was also missing, bonus fix). - `webui/static/helper.js` — WHATS_NEW entry under '2.4.2' dev cycle describing the user-visible change in the chill terse voice. Tests: - `tests/test_download_orchestrator_soundcloud.py` — 14 integration tests verifying the wiring: client constructed at startup, _client lookup resolves 'soundcloud', get_source_status includes it, download dispatcher routes username='soundcloud' to the SoundCloud client (and unknown usernames still fall back to Soulseek), hybrid search iterates SoundCloud when in order and skips it cleanly when unconfigured, get_all_downloads / get_download_status / cancel / clear walk SoundCloud, soundcloud-only mode dispatches only to SoundCloud, _streaming_sources tuple in validation includes 'soundcloud'. - `tests/downloads/test_download_orchestrator.py` — added `soundcloud` to the test fixture's _build_orchestrator helper so the new orchestrator attribute doesn't AttributeError in pre- existing tests that bypass __init__. Verified: - Full suite green (1728 passed, 2 deselected for soundcloud_live) - Ruff clean - Live SoundCloud-only mode search returns 25 SoundCloud tracks for "kendrick lamar luther" in <2s, returning properly-shaped TrackResult objects with username='soundcloud' and dispatch-key filename ready for the download path. Out of scope (intentional deferrals): - SoundCloud Go+ OAuth tier (256 kbps AAC) — anonymous-only for now. Adding auth later is a settings-page extension, no orchestrator changes needed. - Album/playlist support — SoundCloud has playlists but they don't map to the album model the rest of SoulSync expects. Singles only.	4 weeks ago
Broque Thomas	583c4f1e49	Build SoundCloud download client (not yet wired into app) Discord request (Toasti): some tracks (DJ mixes, sets, removed Spotify content) only live on SoundCloud. Add SoundCloud as an option for the existing multi-source download dispatch. This commit only ships the client + tests. Integration into the search dispatch / settings UI / web_server.py routes is intentionally deferred to a follow-up PR — the user-requested workflow is build-and-verify in isolation first, then wire up. `core/soundcloud_client.py`: - SoundcloudClient class mirrors the public surface of every other download client (TidalDownloadClient, QobuzClient, HiFiClient, DeezerDownloadClient): __init__(download_path), set_shutdown_check, is_available / is_configured / is_authenticated, async check_connection, async search returning (List[TrackResult], List[AlbumResult]), async download returning a download_id, _download_thread_worker / _download_sync / _update_download_progress, async get_all_downloads / get_download_status / cancel_download / clear_all_completed_downloads. - Underlying lib: yt-dlp (already in requirements.txt as 2026.3.17). - Anonymous-only — public SoundCloud tracks at the cap quality (typically 128 kbps MP3, occasionally 256 kbps AAC depending on the upload). No FLAC ever; SoundCloud doesn't expose lossless. OAuth tier for SoundCloud Go+ is documented in the module header as a future tier. - Returns standard TrackResult / DownloadStatus dataclasses from core.soulseek_client so downstream matching/post-processing stays source-agnostic. - Filename dispatch key encodes track_id + permalink_url + display_name so the download worker has everything without re-querying. - Heuristic "Artist - Title" parser handles SoundCloud uploaders' typical title format; falls back to uploader handle as artist when the title doesn't have a separator. - Defensive: search returns empty on bad input, missing yt-dlp, or any raised exception. Download sync rejects files under 100KB (preview snippets / broken responses) and cleans them up. - Cooperative cancellation via shutdown_check inside yt-dlp's progress_hooks. Cancelled state survives the download thread's terminal-state assignment. `tests/test_soundcloud_client.py`: - 37 unit tests with yt-dlp stubbed: search shape correctness, the artist/title heuristic, the dispatch-key roundtrip, the download state machine (success / failure / shutdown / Cancelled-state preservation), the progress emitter (progress capping, time remaining), defensive paths (missing yt-dlp, raising yt-dlp, malformed entries, empty entries), and the cancel/clear ledger operations. - 2 live integration tests gated behind `-m soundcloud_live` so CI doesn't run them by default. Run locally with: python -m pytest tests/test_soundcloud_client.py -m soundcloud_live -v - All 37 unit tests pass; both live tests pass against real SoundCloud. - Verified end-to-end with a real album download (Kendrick GNX, 12/12 tracks, 4-7 MB each, completed under 60s per track). `pyproject.toml`: - Register the `soundcloud_live` pytest marker so the unknown-mark warning is suppressed and the live tests can be cleanly gated. Not changed: web_server.py, settings UI, search dispatch, matching engine, WHATS_NEW. Integration is the next PR.	4 weeks ago
Broque Thomas	d8437c87c6	Fix Album Completeness Auto-Fill on Docker / shared-library setups (#476 ) GitHub issue #476 (gabistek, Docker on Arch host): "Auto-Fill" / "Fix Selected" on the Album Completeness findings page returned "Could not determine album folder from existing tracks" for every album. Reproduces on any setup where the media-server library lives outside the SoulSync transfer/download folders — Docker is the headline case but native installs that point Plex at a NAS via SMB hit it too. Root cause: `core/repair_worker.py:_resolve_file_path` only probed the transfer + download folders. Docker users have their Plex/Jellyfin library bind-mounted at /music (or similar) — neither configured in SoulSync. Every existing track got silently treated as missing, so `album_folder` stayed None and the fix workflow bailed. The same incomplete logic was duplicated four more times in the repair_jobs/ modules, all with the same bug. Album Completeness was just the most user-visible — the same setups were also producing false "missing file" findings from Dead File Cleaner, silent skips in MBID Mismatch Detector, etc. The web server already had the correct logic at `web_server.py:_resolve_library_file_path` (probes transfer + download + Plex-reported library locations + user-configured library.music_paths). The repair workers had never been updated to match. Fix: - New `core/library/path_resolver.py` extracts the union logic into a single shared function `resolve_library_file_path()`. Probes (in order, deduped): explicit transfer/download kwargs, config-derived soulseek.transfer_path/download_path, Plex-reported library locations (when a plex_client is passed), user-configured library.music_paths. Each defensive: malformed config or a flaky Plex client degrades to the dirs that did succeed. - `core/repair_worker.py:_resolve_file_path` becomes a delegating wrapper preserving the legacy signature, with a new `config_manager` kwarg. All 15 in-tree call sites updated to thread `self._config_manager` through. - `core/repair_jobs/dead_file_cleaner.py`, `mbid_mismatch_detector.py`, and `lossy_converter.py` get the same treatment: duplicate function replaced with a thin wrapper, call sites pass `context.config_manager`. - `core/repair_jobs/acoustid_scanner.py` and `unknown_artist_fixer.py` (which used to import from repair_worker) now call the shared resolver directly with `context.config_manager`. Side benefit: every other repair job (Dead File Cleaner, MBID Mismatch Detector, Lossy Converter, AcoustID Scanner, Unknown Artist Fixer) also stops missing files in the media-server library mount. Single fix unblocks five user-visible features. Tests: `tests/library/test_path_resolver.py` — 20 cases covering all four base-dir sources, suffix-walk algorithm, dedup, defensive paths (None plex client, malformed config entries, raising config_manager.get, broken plex attribute access), Docker path translation. Full suite 1677 passed locally. WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	42f3026eef	Reject broken downloads before tagging via universal integrity check Discord report (fresh.dumbledore [VRN]): slskd sometimes ships broken files (truncated transfers, corrupt FLAC, wrong file substituted on filename match). They flowed through post-processing and only surfaced later — Plex/Jellyfin scan failures, dead-air playback, duplicate detector tripping over the wrong length. By that point the file was already tagged, copied, mirrored to the media server, and recorded in provenance. New module `core/imports/file_integrity.py`: - `check_audio_integrity(path, expected_duration_ms=None) -> IntegrityResult` - Three tiered checks, cheapest to most expensive: 1. File size sanity (catches 0-byte stubs and stub transfers) 2. Mutagen parse (catches header damage, wrong-format-with-right-extension) 3. Duration agreement vs. metadata source's expected length, ±3s tolerance (5s for tracks over 10 minutes — long tracks naturally drift more) - Returns IntegrityResult with `ok`, human-readable `reason`, and per-check `checks` dict for debugging - Never raises; pathological inputs return ok=False with explanation Pipeline integration in `core/imports/pipeline.py:post_process_matched_download`: - Hooks between the existing file-stability wait and AcoustID verification - On failure: quarantine via existing `move_to_quarantine` helper, mark task failed with descriptive error, clear matched-context, fire `on_download_completed(success=False)` so the slot is released for retry - Mirrors the existing AcoustID-failure path so retry behavior stays consistent - Wrapped in try/except so an unexpected failure inside the check itself cannot block downloads — logs and continues This is intentionally tier 1: universal across formats, no external deps. A future tier could verify FLAC STREAMINFO MD5 by decoding audio (needs flac binary or libflac wrapper) — skipped for now since tier 1 catches the dominant Discord-reported cases (truncated, 0-byte, wrong file). Tests: - `tests/imports/test_file_integrity.py` — 14 cases covering all three check tiers, edge cases (zero/negative expected duration, long-track wider tolerance, caller tolerance override), and the mutagen-unavailable degradation path - `tests/imports/test_import_pipeline.py` — two existing tests use 5-byte fixture files that the new check would reject; they monkeypatch the integrity check since they're testing plumbing (notification + metadata_runtime forwarding), not integrity behavior WHATS_NEW entry under '2.4.2' dev cycle.	4 weeks ago
Broque Thomas	783c543c3e	Auto-import: live per-track progress + in-progress history row User reported (Mushy / generally) that dropping an album into the staging folder left the auto-import history blank for the entire processing window — sometimes 5+ minutes for a full album. Pre- existing UX gap, not caused by the recent context-builder refactor. Two root causes: 1. ``_record_result`` only fired AFTER ``_process_matches`` returned. For a 14-track album with ~30s/track post-processing, that meant ~7 minutes of zero rows in auto_import_history → nothing for ``/api/auto-import/results`` to return → empty UI. 2. ``_current_status`` only ever transitioned between 'idle' and 'scanning' — never 'processing'. ``get_status()`` had no per- track index/name fields, so the UI had no way to render "Processing track 3/14: Mine" even if it wanted to. Fix: - New ``_record_in_progress`` inserts a status='processing' row up-front (before the per-track loop starts) so the UI sees the import the moment it begins. Returns the row id. - New ``_finalize_result`` updates that same row with the final outcome (completed/failed) when processing finishes. One row per album, not per track — keeps the history list clean. - Both share ``_serialize_match_data`` (extracted from the original ``_record_result``) so the in-progress row carries the same match payload shape the existing review UI already understands. - ``_process_matches`` updates ``_current_track_index``, ``_current_track_total``, and ``_current_track_name`` BEFORE each per-track callback fires, so a polling UI sees consistent "processing N/M: <name>" snapshots. - ``_scan_cycle`` flips ``_current_status`` to 'processing' before the per-album loop, resets it + the per-track fields after. Defensive ``finally`` clears progress even if the inner code path raised. - ``get_status()`` exposes the new fields so the UI's existing /api/auto-import/status polling picks them up. - Frontend (stats-automations.js): renders the new ``current_status='processing'`` state with track index/total/name in the existing progress bar element. New 'processing' status class for styling parity with 'scanning'. 8 regression tests in tests/imports/test_auto_import_live_progress.py: - get_status surfaces the new fields with sane defaults - track_index advances 1, 2, 3 during a 3-track loop - track_total set BEFORE the first callback fires (no '1/0' flicker) - _record_in_progress writes status='processing' with no processed_at - _finalize_result updates the same row to completed + processed_at, no second insert - _finalize_result with failed status leaves processed_at NULL - _finalize_result with row_id=None is a safe no-op - Per-track fields cleared by _scan_cycle's finally block Full pytest 1643 passed; ruff clean.	4 weeks ago
Broque Thomas	29089b35b3	Honor configured Tidal redirect_uri, drop request-host fallback Reported case (Foxxify): Tidal returned error 1002 ("Invalid redirect URI") on every authentication attempt for users accessing SoulSync from a network IP. User had ``http://127.0.0.1:8889/tidal/callback`` registered in his Tidal Developer Portal — matching the SoulSync UI default and docs. Root cause: the /auth/tidal route at web_server.py:5594-5598 had a "fallback: dynamically set based on request host" branch that fired when ``tidal.redirect_uri`` config was empty AND the request didn't come from localhost. That fallback overrode the TidalClient constructor's safe default (``http://127.0.0.1:<port>/tidal/callback``) with a uri built from request.host like ``http://192.168.x.x:8889/tidal/callback``. Tidal compares strings exactly so this never matched the documented portal registration and the user got 1002 before the consent screen even rendered. The trap is the SoulSync settings UI displays the default URI as the placeholder + "Current Redirect URI" display — but the placeholder never gets saved to config unless the user explicitly clicks Save. Most users who follow the docs (register the displayed default with Tidal, then click Authenticate) hit the empty-config path and the broken fallback. Fix: drop the request-host fallback. Empty config falls back to the constructor default that matches the documented portal registration. The existing post-auth swap-step in the instructions page below handles the Docker / remote-access case as designed: 1. SoulSync sends 127.0.0.1:8889 in the authorize URL → matches portal → Tidal accepts. 2. User authorizes → Tidal redirects browser to 127.0.0.1:8889 (which fails locally — nothing on user's machine listens there). 3. Instructions tell user to swap 127.0.0.1 with the host they're accessing SoulSync from. 4. Swapped URL hits the container's exposed callback port → auth completes. 8 regression tests in tests/test_tidal_auth_redirect_uri.py: - Configured redirect_uri sent verbatim (localhost / custom port / explicit network IP) - Empty config falls back to constructor default — NOT request.host (the actual reported scenario, with explicit assertion message warning if the bug returns) - Empty config + localhost access uses the same default (sanity) Full pytest 1635 passed; ruff clean.	4 weeks ago
Broque Thomas	24c2d75c6d	Make extract_external_ids recognize all source-tagging conventions Smoke-testing the just-merged provenance PR against live logs revealed the new ID-match block was silently no-opping: no [ExtID Match] / [Provenance Match] log lines despite the code path being live. Tracing revealed two related gaps in extract_external_ids' source detection: 1. Underscore-prefixed key. Deezer / Discogs / Hydrabase clients tag normalized track dicts with ``_source`` (underscore prefix — convention used in 8+ places across core/). The extractor only looked for ``provider`` and ``source``, so Deezer-sourced tracks silently returned no IDs. 2. No provider field at all. Spotify and iTunes raw API responses carry ``id`` but no provider/source key of any kind. The extractor couldn't disambiguate the native ``id``, so Spotify-primary scans would have hit the same silent miss once the user switched primary sources. Two-part fix: - ``extract_external_ids`` now recognizes ``_source`` as another candidate provider field. - New optional ``source_hint`` parameter lets the caller supply the configured primary source as a fallback when the track dict has no provider field of its own. Track-side provider field still wins when present (defensive against a wrong hint). Watchlist scanner now passes ``get_primary_source()`` as the hint so both naming conventions (Deezer-style _source, Spotify-style no-tag) get handled uniformly. 6 new regression tests cover: - _source recognized for Deezer - _source recognized for Hydrabase (cross-provider mapping) - _source recognized for Discogs (no library column — verifies graceful no-crash) - source_hint disambiguates raw tracks for spotify/itunes/deezer - track-side provider takes precedence over hint - None hint defaults safely Full pytest 1630 passed; ruff clean. After this lands and the server restarts, watchlist scans should produce [ExtID Match] / [Provenance Match] log lines for tracks already on disk regardless of which metadata source the user has configured as primary.	4 weeks ago
Broque Thomas	34ba26f5c8	Persist source IDs at download time + backfill onto tracks on sync Followup to fix/watchlist-external-id-match. The companion PR closed the demand side — the watchlist scanner asks for tracks by external IDs before falling back to fuzzy. But for users on Plex / Jellyfin / Navidrome the supply side was still broken: tracks.spotify_track_id (and the other ID columns) only got populated by the asynchronous enrichment workers, sometimes hours after the file was actually written. During that window the ID match fell through to fuzzy and the bug returned. We were already collecting every ID during post-processing — they live in the `pp` dict in core/metadata/source.py:embed_source_ids and get embedded into file tags. We just dropped the in-memory copy afterwards. This PR persists them and uses them: - Schema migration adds spotify_track_id / itunes_track_id / deezer_track_id / tidal_track_id / qobuz_track_id / musicbrainz_recording_id / audiodb_id / soul_id / isrc columns + indexes to the existing track_downloads table (already keyed by file_path). - core/metadata/source.py:embed_source_ids exposes pp["id_tags"] and the resolved ISRC back to the import context as _embedded_id_tags / _isrc. - core/imports/side_effects.py:record_download_provenance reads those context fields and passes them to db.record_track_download, which now accepts the new ID kwargs and persists them. - New db.get_provenance_by_file_path with exact + basename-suffix fallback (handles container mount-root differences between download-time path and media-server-reported path). - New db.backfill_track_external_ids_from_provenance copies IDs from track_downloads onto a tracks row idempotently — COALESCE on every column preserves any value the enrichment worker already wrote (enrichment is more authoritative for late binding). - database/music_database.py:insert_or_update_media_track (the single insertion point used by every Plex / Jellyfin / Navidrome sync) calls the backfill immediately after each INSERT/UPDATE. - New core/library/track_identity.py:find_provenance_by_external_id used as a second-tier fallback in watchlist_scanner.is_track_missing _from_library — catches the window between download and media-server sync. Caller checks os.path.exists on the provenance file_path before treating it as "already in library" so a deleted file doesn't prevent re-download. Effect: freshly downloaded files become ID-recognizable to the watchlist on the very next scan, no enrichment-wait window. 19 regression tests in tests/test_provenance_id_persistence.py: - Schema migration adds expected columns + indexes - record_track_download persists every ID kwarg - record_track_download backward-compat (old kwargs still work) - get_provenance_by_file_path: exact match, basename fallback for mount-root differences, multi-record latest-wins, defensive None - backfill: copies all IDs, preserves existing via COALESCE, no-op when no provenance exists - find_provenance_by_external_id: per-ID lookup, ISRC cross-bridge, OR semantics, latest-wins on multiple matches Out of scope: backfilling provenance for files downloaded BEFORE this PR (their track_downloads rows don't carry the new IDs). Those continue to wait for enrichment. Acceptable — only affects historical files; new downloads benefit immediately. Full pytest 1625 passed; ruff clean.	4 weeks ago
Broque Thomas	ecb8939c80	Match library tracks by external IDs before fuzzy in watchlist scan Reported case (CAL): a track already on disk got re-downloaded by the watchlist scanner on every scan. Library DB had stale album metadata for the file (track tagged on album "Left Alone") while the metadata source reported it on a different album ("NPC" single). The title+artist+album fuzzy block correctly said the album names didn't match and declared the track missing — but the file's stable external IDs (Spotify ID, ISRC, etc.) unambiguously identified it as the same recording. The earlier compilation-album fix (PR #461) handled qualifier drift ("OST" vs "Music From The Motion Picture"). This case is two genuinely different album names referring to the same song. Fix: provider-neutral external-ID short-circuit before the fuzzy block in `is_track_missing_from_library`. Pulls every recognized ID off the source track (Spotify / iTunes / Deezer / Tidal / Qobuz / MusicBrainz / AudioDB / Hydrabase / ISRC), runs a single SELECT against the indexed external-ID columns on the `tracks` table, and treats any hit as "track exists in library — don't re-download". If no IDs are available (older imports without enrichment, library scans that didn't populate external IDs), falls through to the existing fuzzy logic so the safety net stays intact. New `core/library/track_identity.py` module with two helpers: - `extract_external_ids(track)`: handles dict and object-style track shapes, direct-field aliases (spotify_id / spotify_track_id / SPOTIFY_TRACK_ID), and provider-disambiguated native `id` fields (when track has `provider='deezer'` and `id='X'`, treats X as a Deezer ID). - `find_library_track_by_external_id(db, external_ids, server_source)`: builds an OR of indexed column matches with IS NOT NULL guards, optional server_source filter that also passes legacy NULL rows, single-row LIMIT. ISRC bridges across providers — a library track imported via Deezer can be matched against a Spotify scan when both sides carry the same ISRC. 43 regression tests in `tests/test_library_track_identity.py`: - 9 ID-extraction tests for direct fields (Spotify / iTunes / Deezer / ISRC / MBID / AudioDB / Hydrabase) - 8 ID-extraction tests via the provider field (8 providers + source alias + missing-provider-ignored) - 7 mixed/defensive tests (multiple IDs, object-style, empty strings, None track, numeric coercion) - 8 lookup tests (per-provider + ISRC cross-bridge) - 3 OR-semantics tests - 4 server_source filter tests - 2 ID-column-map sanity tests Full pytest 1606 passed; ruff clean.	4 weeks ago
Broque Thomas	486116c34f	Honor lossy_copy.delete_original after successful conversion Reported case (CAL): with lossy_copy.enabled=True, lossy_copy.delete_original=True, and codec=mp3, every download left both the original FLAC AND the converted MP3 in the target folder. Users opting into a lossy-only library ended up dual-format on every import. Root cause: ``core/imports/file_ops.py:create_lossy_copy`` reads ``lossy_copy.codec`` and ``lossy_copy.bitrate`` from config but never reads ``lossy_copy.delete_original``. The setting is only consulted by the pre-move source-vanished check at ``core/imports/pipeline.py:651`` (so the pipeline knows to look for a lossy variant when the FLAC has already moved on), but no code path actually deletes the source after conversion. Fix: after ffmpeg returns success and the QUALITY tag is written, check ``lossy_copy.delete_original`` and ``os.remove`` the original when enabled. Belt-and-suspenders: - Same-path guard (``os.path.normpath(out_path) != os.path.normpath(final_path)``) prevents accidentally wiping the just-converted file if a future codec choice somehow resolves out_path to the source path. - ``FileNotFoundError`` is treated as success (concurrent worker / dedup cleanup got there first). - Other ``OSError`` (permission denied, locked file) is logged but doesn't propagate — the conversion already succeeded, the user just has to clean up the original manually. Failure paths skip the delete: - ffmpeg returns non-zero → returns None, original stays - lossy_copy.enabled=False → early return before conversion runs - delete_original=False (default) → original stays 7 regression tests cover honored-when-enabled, kept-when-disabled, default-keep, ffmpeg-failure-path, lossy-disabled-path, racing-delete, and locked-file paths. Full pytest 1563 passed; ruff clean. Note: this PR does NOT address the second bug CAL mentioned (track re-downloaded despite already existing on disk). That symptom is caused by stale album metadata on the user's existing files — the library DB has the track tagged on a different album than the metadata source reports — combined with wishlist.allow_duplicate_tracks defaulting to True. Same class of issue partially addressed in PR fix/watchlist-redownload-and-duplicate-detection but compilation- album drift is the only currently-handled case. Tracking separately.	4 weeks ago
Broque Thomas	99dbe265de	Sync Qobuz auth to enrichment worker after login Discord-reported (Foxxify): logging in to Qobuz via the Connect button on Settings showed "Connected: <username> (Active)" but underneath an error said "Qobuz not authenticated...", and the dashboard indicator stayed yellow. Saving settings or reloading the tab didn't help. Root cause: SoulSync runs two QobuzClient instances side by side — one through soulseek_client.qobuz for the /api/qobuz/auth/* endpoints, and a second owned by the enrichment worker thread for thread safety. The login flow only updated the auth-flow instance's in-memory state (plus persisted to config). The dashboard's "configured" check at web_server.py:3371 reads ``qobuz_enrichment_worker.client.user_auth_token`` — the WORKER's instance — which still believed itself unauthenticated. The connection-test step at core/connection_test.py:370 hits the same worker instance for the same reason. Fix: add ``QobuzClient.reload_credentials()`` — a public, network-free method that re-reads the saved session from config and updates the instance's in-memory state + session headers. Call it on the enrichment worker's client immediately after a successful ``/api/qobuz/auth/login``, ``/api/qobuz/auth/token``, or ``/api/qobuz/auth/logout`` so the two instances stay in lockstep without waiting for the next process restart. Unlike the existing ``_restore_session()`` this skips the network probe — the caller has just authenticated, so the token is known good. A small ``_sync_qobuz_credentials_to_worker()`` helper in web_server.py wraps the call so all three endpoints share one path. 10 new regression tests cover the populate / clear / partial-config paths plus the actual two-instance-sync scenario from the bug report. Full pytest 1555 passed (the one pre-existing flake in test_tidal_auth_instructions.py is order-dependent and unrelated).	4 weeks ago
Antti Kettunen	b85a05fb88	Move image URL normalization into metadata helpers - keep existing /api/image-proxy URLs from being wrapped again - reuse the shared metadata package instead of duplicating URL logic in web_server.py - add regression coverage for proxy passthrough and internal URL normalization	4 weeks ago

1 2 3 4 5

235 Commits (6a75d656fa44763edf8caa04937f954bfb4a116f)