mirror of https://github.com/Nezreka/SoulSync.git
main
dev
video
fix/disable-beatport-features
johnbaumb-discover-redesign
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2.0
2.1
2.2
2.3
2.4.0
2.4.1
2.4.2
2.5.0
2.5.1
2.5.2
2.5.3
2.5.4
2.5.5
2.5.6
2.5.7
2.5.9
2.6.0
2.6.1
2.6.2
2.6.3
2.6.4
2.6.5
2.6.6
2.6.7
2.6.8
2.6.9
2.7.0
2.7.1
2.7.2
2.7.3
2.7.4
2.7.5
2.7.6
2.7.7
2.7.8
2.7.9
v0.65
${ noResults }
21 Commits (8ffdca363642be5f40bbefad8a2b9dbb41786c8c)
| Author | SHA1 | Message | Date |
|---|---|---|---|
Francesco Durighetto
|
9ff2e7084a |
Fix organize-by-playlist downloads: library entries, wishlist, and stale Spotify cache
Persist organize_by_playlist on mirrored playlists and run playlist-folder downloads from the auto-sync pipeline instead of the global wishlist phase. Register SoulSync library rows after playlist-folder post-processing, route failed organize batches to the wishlist correctly, and skip sync-time unmatched wishlist only when organize download handles retries. Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL), re-mirror on refetch, and improve standalone playlist modals (re-analysis, Open in Mirrored). Add filesystem missing-track detection and tests. Co-authored-by: Cursor <cursoragent@cursor.com> |
3 weeks ago |
BoulderBadgeDad
|
cea897cbd1 |
Wishlist: import Optional (fix ruff F821 in processing.py)
make_wishlist_batch_row / _run_wishlist_cycle annotate params with Optional, but the typing import only had Any/Callable/Dict. Slipped past py_compile + tests because 'from __future__ import annotations' makes annotations strings (never evaluated at runtime), but ruff flags it statically (F821). |
4 weeks ago |
|
BoulderBadgeDad
|
d3c897fb9d |
Wishlist: route the manual flow through the shared engine (manual == auto)
Stage 2: the manual 'Download Wishlist' flow now calls the same _run_wishlist_cycle engine the auto timer uses, so a manual scan runs the exact same code path as an auto scan. The old bespoke manual orchestration (build payloads + SERIAL inline dispatch) is deleted — its grouping/dispatch was a near-duplicate of auto's that had already drifted. Behavior changes (all intended, discussed): - Manual now dispatches album bundles in PARALLEL (album pool) like auto, instead of serially on one thread. A single cycle='albums' engine call covers the whole selection (albums bundled, singles/ungroupable -> per-track residual), so no 'both cycles' pass is needed. - The manual placeholder batch_id is reused as the engine's first sub-batch (first_batch_id), so the modal's existing poll target stays valid. - WishlistManualDownloadRuntime gains album_bundle_executor (wired in web_server, falls back to the shared pool when unset). - 'Don't start manual while auto is running' is unchanged — the existing route guard (is_wishlist_actually_processing -> 409) already covers it; no queue added. NOT touched: process_wishlist_automatically's behavior (proven by test_automation staying green in Stage 1) and the per-track download mechanics. test_manual_download.py rewritten to characterize the new behavior (engine dispatch via the executor, parallel, placeholder reuse, album-context). Full wishlist suite green (131); wishlist + automation = 392 passed. |
4 weeks ago |
|
BoulderBadgeDad
|
db1e51109c |
Wishlist: extract shared _run_wishlist_cycle engine; auto delegates to it
Stage 1 of unifying the auto + manual wishlist flows. Extract the group -> per-album+residual batches -> register -> dispatch logic that lived inline in process_wishlist_automatically into a standalone _run_wishlist_cycle() engine (built on make_wishlist_batch_row). The auto path now just calls it. Per-flow differences are arguments (auto_initiated stamps the auto-only fields + selects auto vs manual naming/logging; first_batch_id lets a caller reuse a pre-created placeholder). Album batches dispatch to the dedicated album pool, residual to the shared pool (unchanged from #740). Auto behavior is PROVABLY unchanged: its full characterization suite (test_automation.py) stays green (10/10), and the whole wishlist suite passes (131). This commit does NOT touch the manual flow yet (Stage 2) and does not change what auto does — it only moves auto's logic behind a shared entrypoint the manual flow will call next. |
4 weeks ago |
|
BoulderBadgeDad
|
e4b5cbbe60 |
Wishlist: unify batch-row construction into make_wishlist_batch_row
The auto and manual wishlist flows each built the same ~20-field
download_batches row in separate places (auto album, auto residual, manual
placeholder, manual sub-batches) — four near-identical literals that could (and
did) drift apart, producing subtly different batch shapes between the flows.
Extract make_wishlist_batch_row() as the single source of truth: it emits the
consistent core field set, with the genuinely per-flow differences as explicit
arguments — initial phase ('queued' for auto / 'analysis' for manual), the
auto-only auto_initiated/auto_processing_timestamp/current_cycle via
extra_fields, and album-vs-residual contexts. All four sites now go through it,
so every wishlist batch has an IDENTICAL shape (this also removes the field
drift that confused the modal-hydration code).
Deliberately NOT unified — and left explicit in each caller, per the
'don't cargo-cult genuinely-different code' principle: the grouping decision
(auto groups only on the albums cycle), batch-id allocation (manual reuses the
caller's placeholder id for the first sub-batch), and dispatch (auto
parallel-submits album batches to the dedicated pool + residual to the shared
pool; manual runs them serially on one thread). Those are real behavioral
differences, not duplication.
Behavior-preserving: verified safe to normalize the row shape (grep confirmed
every reader uses .get() with defaults, no key-presence checks). The existing
auto (test_automation.py) and manual (test_manual_download.py) characterization
suites stay green = differential proof of identical behavior. Adds
test_batch_factory.py (core fields, album/residual, extra_fields, no shared
mutable state, consistent key shape). 131 wishlist tests pass.
|
4 weeks ago |
|
BoulderBadgeDad
|
0898014364 |
Fix #740: run wishlist album-bundle downloads on a dedicated pool
A 2.6.3 change (
|
4 weeks ago |
Broque Thomas
|
6841128dc2 |
Wishlist: distinguish Queued from Analyzing for executor-pending batches
PR 4 of 4 in the wishlist-album-bundle issue series. UI fix only —
zero behavior change.
User's 26-track wishlist run rendered all 26 sub-batches as
"Analyzing..." simultaneously. Pre-fix the rows were created with
``phase='analysis'`` BEFORE being submitted to ``missing_download_executor``
(max_workers=3 by default), so 23 batches sat in the executor queue
visually identical to the 3 actually running. Misled users into
thinking SoulSync was processing 26 in parallel; really only 3 ever
ran at once with the rest waiting their turn.
Fix:
- Wishlist auto-flow submission sites now create batch rows with
``phase='queued'``.
- The master worker (``core/downloads/master.py:328``) already flipped
phase to ``'analysis'`` as its first action on entry — that
transition becomes the real signal that the executor picked the
batch up.
- ``core/downloads/status.py`` surfaces ``analysis_progress`` for
the ``queued`` phase too so the UI has the track count to render
"Queued — N tracks" instead of an empty card.
- Frontend (``webui/static/pages-extra.js``, ``downloads.js``) renders
"Queued ⏳" for ``phase='queued'`` distinct from the spinner-laden
"Analyzing..." for ``phase='analysis'``.
Scope choices:
- Only the auto-wishlist submission sites flipped this PR
(``core/wishlist/processing.py:860`` album sub-batches +
``core/wishlist/processing.py:907`` residual). The manual-wishlist
sites at ``:451`` and ``:627`` use the same executor + worker, but
those create a caller-allocated batch_id that the frontend polls
immediately — wanted to verify the manual-poll path handles
``queued`` cleanly before flipping those. Trivial follow-up.
- Other submission sites in album_bundle_dispatch / web_server.py /
task_worker.py left untouched — they don't go through the
executor-queue pattern that causes this UI confusion.
Tests:
- Updated ``test_process_wishlist_automatically_creates_batch_for_matching_tracks``
to assert ``phase='queued'`` on creation (was ``'analysis'``); explanatory
comment names the executor-pool reason.
- New ``test_queued_phase_surfaces_analysis_progress_for_ui_count`` in
``tests/downloads/test_downloads_status.py`` pinning the new
``queued ⊂ analysis_progress`` rendering contract.
- 884 tests pass across wishlist + downloads + imports suites.
- Ruff clean on changed Python files; JS syntax OK on changed
webui files.
PR 3 (sibling-completion gate) was investigated and dropped — the
"1/26 finalized" symptom turns out to be downstream of the
staging-match bug (PR 2's instrumentation will catch it on the
user's next reproduction run), not an independent sibling-gate bug.
The gate logic itself is correct.
|
4 weeks ago |
|
Broque Thomas
|
dd32e3bbe1 |
Wishlist: only engage album-bundle when multiple tracks from same album (PR 1/4)
Real-world wishlist case the original
|
4 weeks ago |
|
Broque Thomas
|
c002014f10 |
Wishlist: reify run id + gate cycle toggle on last-sibling completion
Phase 1c.2.1 splits each wishlist invocation into per-album sub- batches so the album-bundle dispatch can engage once per album. Side effect: the completion handler ``finalize_auto_wishlist_completion`` ran end-of-run logic (cycle toggle + state reset + automation event emit) once per BATCH, so a 2-album run fired the cycle toggle twice + emitted two ``wishlist_processing_completed`` events. The cycle landed at the right value either way but the state machine had become per-batch instead of per-run. Fix: reify "wishlist run" as a first-class concept via a shared ``wishlist_run_id`` UUID. Generated once per wishlist invocation in both the auto- and manual-wishlist paths, stamped on every sub-batch row in ``download_batches``. ``finalize_auto_wishlist_completion`` now reads the completing batch's ``wishlist_run_id`` and, when present, scans ``download_batches`` for siblings still in pre-terminal phases. If any sibling is still active, the per-batch summary records but the cycle toggle + state reset + automation emit are deferred. Only the last completing sibling fires the run-level finalization. Legacy single-batch runs (no run_id field) keep their toggle-immediately behavior — back-compat by absence. The run_id also lays groundwork for frontend grouping (one logical row in the Downloads view per wishlist run instead of N sibling rows), but that UX work is deferred. 3 new tests in ``test_processing.py`` pin: defer-when-siblings- active, toggle-when-last-sibling-done, back-compat-without-run_id. 1 new assertion in ``test_automation.py`` confirms all sub-batches of one auto-wishlist invocation share the same run_id. 309 tests across wishlist + automation suites green. Notes: dispatch concurrency unchanged — sub-batches still run via the shared download worker pool. Slskd serializes per-uploader at its own layer (same uploader = automatic queue, different uploaders = legit parallel), so SoulSync-side serial enforcement would duplicate work the right layer already handles. |
4 weeks ago |
|
Broque Thomas
|
7832acba31 |
Manual wishlist run: also split into per-album sub-batches
The Phase-1 fix (commit
|
4 weeks ago |
|
Broque Thomas
|
c3b88e6963 |
Wishlist albums cycle: split into per-album bundle batches
Auto-wishlist's "albums" cycle used to dump every missing album track into one batch and run per-track Soulseek / Prowlarr searches for each (~50 searches for a typical scan). The album-bundle dispatch (introduced in 2.5.9 for explicit album downloads) was gated on ``is_album_download=True`` + populated ``album_context``/``artist_context``, none of which the wishlist batch ever set — so wishlist runs always took the per-track flow even when 12 missing tracks all belonged to the same album. Fix: split wishlist albums-cycle tracks into per-album sub-batches at submission time. Each sub-batch carries its own album context, trips the existing dispatch gate, and engages one slskd / torrent / usenet album-bundle search per album. Tracks the helper can't group (no album metadata, no artist) fall through to a residual per-track batch. - New ``core/wishlist/album_grouping.py``: ``group_wishlist_tracks_by_album(tracks)`` returns ``WishlistGroupingResult(album_groups, residual_tracks)``. Pure function — extracts album_id (or name-normalized fallback) + primary artist + album context from each track's nested spotify_data, buckets, and threshold-promotes. Independent of runtime state so it can be unit-tested without the wishlist executor. - ``core/wishlist/processing.py``: when ``current_cycle == 'albums'``, run the grouping helper, submit one batch per album with ``is_album_download=True`` + the group's album/artist context, then a single residual batch for orphans. Singles cycle path unchanged. - 9 new tests in ``test_album_grouping.py`` pin the bucketing contract (empty / single album / multi album / orphan / threshold / nested payloads / no-id fallback / no artist). - 2 new tests in ``test_automation.py`` exercise the per-album split end-to-end through ``process_wishlist_automatically``: multi-album batch → two sub-batches each with album context; mixed orphan + real album → one bundle batch + one residual. 1099 tests across wishlist + imports + downloads + automation + playlist-sources + staging-provenance + track-number-repair suites green. WHATS_NEW entry added under 2.6.3. Now when an auto-wishlist scan finds 12 missing tracks from Ryoto's "Cha-La Head-Cha-La", it runs ONE slskd / Prowlarr album-bundle search for the release instead of 12 per-track searches. |
4 weeks ago |
|
Broque Thomas
|
85ba93f16f |
Fix album-bundle staging match + wishlist provenance (#700, #698)
Root cause (#700): the Soulseek album-bundle path downloads whole releases into a private staging dir, then per-track workers claim those files via the staging-match shortcut. When slskd files arrived without ID3 tags (common for FLAC rips), the staging cache fell back to the filename stem as the title — and stems shaped like "Artist - Album - 03 - Title" could not clear the 0.80 title- similarity threshold against the clean Spotify track name. Every track in the album went not_found, the batch ended "failed" in the Downloads UI with an empty queue, and the bundle-downloaded files just sat unused in staging. Fix: in _staging_title_variants, add a trailing-title variant by extracting the segments after a bare track-number block (e.g. "03") between " - " delimiters. Conservative — only fires when a clear digit segment is present, so real song titles with dashes like "Hold Me - Live" are left intact. Generated as an additional variant alongside the existing raw/compacted/feat-stripped/bonus-stripped forms, so behavior on already-matching files is unchanged. Downstream (#698): the album-bundle staging miss pushed every failed track to the wishlist labelled as a playlist track, and a couple of fallback paths in ensure_wishlist_track_format and the slskd-result reconstruction hardcoded album_type='single' / total_tracks=1 on the stored album dict. On wishlist requeue the path builder saw album_type='single' and routed the download through single_path, dumping the file in the Singles tree even though it belonged to an album. (Running Reorganize would fix it because the DB album linkage was still correct, but the file landed in the wrong place first.) Fixes: - new resolve_wishlist_source_type_for_batch() returns 'album' for is_album_download batches; wishlist_failed.py now calls it instead of hardcoding 'playlist' - build_wishlist_source_context() threads album_context / artist_context / is_album_download from the batch into the wishlist row so future requeue logic has authoritative routing data - the non-dict-album fallback in ensure_wishlist_track_format and the slskd-result reconstruction default album_type='album' (and total_tracks=0 = unknown) instead of lying with 'single'/1; the existing setdefault chain handles dict-shaped album data unchanged Tests: - 2 staging-match tests pin the new tail-extraction behavior against a realistic untagged slskd stem, plus a negative test that confirms a dash-in-title without a digit segment still does NOT extract a variant - 2 payload tests pin the album_type='album' default for both fallback paths - 4 processing tests pin resolve_wishlist_source_type_for_batch() and the album-context threading in build_wishlist_source_context() 3974 pass; no behavioural change on already-working flows. |
1 month ago |
|
Broque Thomas
|
aaf312cd34 |
Honor manual library matches across source labels
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded. Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path. |
1 month ago |
|
Broque Thomas
|
3e7eeb7c9c |
Honor manual matches in automatic wishlist cleanup
|
1 month ago |
|
Broque Thomas
|
42f4aa5eac |
Add manual library track matching
|
1 month ago |
|
Broque Thomas
|
aa54bed818 |
Surface silent exceptions across remaining modules — ~70 sites
Final sweep. Covers: - Downloads: candidates / lifecycle / master / monitor / wishlist_failed - Metadata: source / registry / cache / common / artwork (+ plex_client) - Imports: pipeline / resolution / file_ops / paths / guards - Library: path_resolver / retag / duplicate_cleaner - Stats / playlists / wishlist / discovery / automation / enrichment - Misc: hydrabase_client, soulsync_client, tag_writer, debug_info, api_call_tracker, album_consistency, beatport_unified_scraper, reorganize_runner, seasonal_discovery, lidarr_download_client, services/sync_service.py, automation_engine, automation/progress Two `_e` renames in imports/file_ops.py (outer scope binding `e`). A few finally-block sites in metadata/album_mbid_cache.py, library/track_identity.py, listening_stats_worker.py, watchlist/ auto_scan.py left silent — same reason as the rest of the sweep (logger calls during cleanup paths can themselves raise). Refs #369 |
2 months ago |
|
Broque Thomas
|
99a763dace |
fix: drop redundant library-cleanup pass from wishlist download flows
Both the auto and manual wishlist download paths called `remove_tracks_already_in_library` before submitting the batch — a serial DB lookup per track per artist (~1s/track on a 24-track wishlist). The batches set `force_download_all=True` which is explicitly documented as "skip the expensive library check" — the pre-flight cleanup was contradicting that flag. Removed the cleanup call from both flows. Kept `remove_wishlist_duplicates` (fast SQL DELETE) and the standalone `/api/wishlist/cleanup` endpoint that exposes the library scan as explicit user-triggered maintenance. Safety check on the trade-off: - post-processing at `core/imports/pipeline.py:576-624` already handles re-downloads defensively: existing file with metadata → skip overwrite + delete source duplicate, no library corruption. - Master worker's analysis loop normally removes wishlist entries for found tracks via `_check_and_remove_track_from_wishlist_by_metadata`, so stale wishlist entries should be rare in practice. - Worst case for the rare orphan: one redundant download attempt that the post-processing layer no-ops on. Bandwidth waste, not data damage. Tests updated: - `..._does_not_run_library_cleanup` (renamed from `_skips_enhance_tracks_during_cleanup`) asserts no DB track-existence checks happen and no wishlist removals fire — both `enhance` and "owned" tracks reach the master worker. - `..._marks_batch_complete_when_wishlist_genuinely_empty` (renamed from `..._after_cleanup`) covers the path where the wishlist starts empty. Full suite: 1232 passing. Ruff clean. |
2 months ago |
|
Broque Thomas
|
6a25dcd49e |
fix: move manual wishlist cleanup into background worker
The manual wishlist download endpoint blocked the request thread on a
slow library-cleanup pass before submitting the batch — for a 24-track
wishlist that's ~50 per-track DB lookups serialised in the request
handler, taking 30+ seconds before the frontend got a response. The
modal sat at "Pending..." with no progress visible the whole time.
Split start_manual_wishlist_download_batch into:
1. SYNC path (request handler):
- Generate batch_id, create download_batches entry with phase=analysis
and analysis_total=0 placeholder.
- Submit a single bg job (`_prepare_and_run_manual_wishlist_batch`) to
the missing-download executor.
- Return 200 with batch_id immediately. Frontend can start polling
/api/active-processes status right away.
2. BG path (executor thread):
- db.remove_wishlist_duplicates (slow-ish, single SQL)
- remove_tracks_already_in_library (the slow one — per-track DB checks)
- wishlist_service.get_wishlist_tracks_for_download
- sanitize + dedupe + filter (track_ids / category)
- Update batch.analysis_total with the real filtered count
- add_activity_item("Wishlist Download Started", ...)
- run_full_missing_tracks_process (master worker)
Edge case: if cleanup empties the wishlist, the bg job marks the batch
phase='complete' with error='No tracks in wishlist' (instead of the old
synchronous 400 response). Frontend status poll picks this up and the
modal can close cleanly.
Tests: existing 2 manual-download tests updated to drive the bg job
explicitly via a new `_run_submitted_bg_job` helper. Added 2 new tests:
- `..._returns_immediately_with_placeholder` — proves the sync path
doesn't trigger any cleanup or master-worker calls; analysis_total=0.
- `..._marks_batch_complete_when_wishlist_empty_after_cleanup` —
cleanup empties the list, master worker never invoked, batch ends
with phase='complete'.
Full suite: 1232 passing (was 1230). Ruff clean.
|
2 months ago |
Antti Kettunen
|
0125f478fc
|
Trim wishlist runtime plumbing
- remove the redundant wishlist-service injection from the runtime wrappers - keep the package owning its own singleton service access - simplify the route runtime API and update the wishlist tests to match |
2 months ago |
|
Antti Kettunen
|
f5226bd5b5
|
Give wishlist modules their own loggers
- add module-level loggers for the wishlist package instead of threading the web server logger through runtime objects - default wishlist helper runtimes and cleanup helpers to their package logger while still allowing test overrides - keep web_server.py as a thin caller that no longer injects its logger into wishlist flows |
2 months ago |
|
Antti Kettunen
|
f32fc9d56e
|
Extract wishlist logic into dedicated package
- add core/wishlist as the home for wishlist payload, resolution, state, processing, reporting, and selection helpers - move wishlist-specific tests into tests/wishlist alongside the new package layout - keep web_server.py and the import/search callers as thin adapters for now |
2 months ago |