Phase 1c.2.1 splits each wishlist invocation into per-album sub-
batches so the album-bundle dispatch can engage once per album.
Side effect: the completion handler ``finalize_auto_wishlist_completion``
ran end-of-run logic (cycle toggle + state reset + automation
event emit) once per BATCH, so a 2-album run fired the cycle
toggle twice + emitted two ``wishlist_processing_completed``
events. The cycle landed at the right value either way but the
state machine had become per-batch instead of per-run.
Fix: reify "wishlist run" as a first-class concept via a shared
``wishlist_run_id`` UUID. Generated once per wishlist invocation
in both the auto- and manual-wishlist paths, stamped on every
sub-batch row in ``download_batches``.
``finalize_auto_wishlist_completion`` now reads the completing
batch's ``wishlist_run_id`` and, when present, scans
``download_batches`` for siblings still in pre-terminal phases.
If any sibling is still active, the per-batch summary records
but the cycle toggle + state reset + automation emit are
deferred. Only the last completing sibling fires the run-level
finalization. Legacy single-batch runs (no run_id field) keep
their toggle-immediately behavior — back-compat by absence.
The run_id also lays groundwork for frontend grouping (one
logical row in the Downloads view per wishlist run instead of N
sibling rows), but that UX work is deferred.
3 new tests in ``test_processing.py`` pin: defer-when-siblings-
active, toggle-when-last-sibling-done, back-compat-without-run_id.
1 new assertion in ``test_automation.py`` confirms all sub-batches
of one auto-wishlist invocation share the same run_id. 309 tests
across wishlist + automation suites green.
Notes: dispatch concurrency unchanged — sub-batches still run via
the shared download worker pool. Slskd serializes per-uploader at
its own layer (same uploader = automatic queue, different
uploaders = legit parallel), so SoulSync-side serial enforcement
would duplicate work the right layer already handles.
The Phase-1 fix (commit c3b88e69) only extended the per-album
bundle dispatch to ``process_wishlist_automatically``. The manual
"Run Wishlist Now" path goes through
``_prepare_and_run_manual_wishlist_batch`` instead, so the
behavior didn't change for users who triggered downloads from the
Wishlist tab UI — they still saw N per-track Soulseek searches
when N missing tracks all came from one album.
Caught in a real-app test: user added Katy Perry's PRISM (Deluxe)
to the wishlist + clicked "Download Wishlist" → app log shows
``_prepare_and_run_manual_wishlist_batch:421`` running a single
batch with 16 tracks + per-track searches firing one by one
("katy perry prism deluxe legendary lovers", "katy perry prism
deluxe roar", etc.), no album-bundle dispatch.
Fix:
- ``_prepare_and_run_manual_wishlist_batch`` now runs the same
``group_wishlist_tracks_by_album`` helper after filtering. For
each detected album, it builds a sub-batch with
``is_album_download=True`` + populated album/artist context.
Residual tracks (no resolvable album metadata) land in a single
per-track residual batch.
- The first sub-batch re-uses the caller-allocated ``batch_id``
so the frontend's existing poll against it keeps working;
additional sub-batches get fresh ids materialized into
``download_batches`` so they show up in the Downloads view.
- Sub-batches dispatch serially — each ``run_full_missing_tracks_process``
call blocks until the album-bundle staging + per-track tasks
complete before the next album's bundle search fires.
New test ``test_manual_wishlist_splits_into_per_album_sub_batches``
pins the contract — multi-album wishlist content with
nested-spotify_data shape produces N master-worker calls (one per
album), each batch carries the album_context, first sub-batch
re-uses the original batch_id. 106 wishlist tests + 1099 across
the broader suite green.
Adding 16 Katy Perry PRISM tracks to wishlist + clicking download
should now fire ONE slskd album-bundle search for the release
instead of 16 individual searches.
Auto-wishlist's "albums" cycle used to dump every missing album
track into one batch and run per-track Soulseek / Prowlarr searches
for each (~50 searches for a typical scan). The album-bundle
dispatch (introduced in 2.5.9 for explicit album downloads) was
gated on ``is_album_download=True`` + populated
``album_context``/``artist_context``, none of which the wishlist
batch ever set — so wishlist runs always took the per-track flow
even when 12 missing tracks all belonged to the same album.
Fix: split wishlist albums-cycle tracks into per-album sub-batches
at submission time. Each sub-batch carries its own album context,
trips the existing dispatch gate, and engages one slskd / torrent
/ usenet album-bundle search per album. Tracks the helper can't
group (no album metadata, no artist) fall through to a residual
per-track batch.
- New ``core/wishlist/album_grouping.py``:
``group_wishlist_tracks_by_album(tracks)`` returns
``WishlistGroupingResult(album_groups, residual_tracks)``.
Pure function — extracts album_id (or name-normalized fallback)
+ primary artist + album context from each track's nested
spotify_data, buckets, and threshold-promotes. Independent of
runtime state so it can be unit-tested without the wishlist
executor.
- ``core/wishlist/processing.py``: when ``current_cycle ==
'albums'``, run the grouping helper, submit one batch per album
with ``is_album_download=True`` + the group's album/artist
context, then a single residual batch for orphans. Singles
cycle path unchanged.
- 9 new tests in ``test_album_grouping.py`` pin the bucketing
contract (empty / single album / multi album / orphan / threshold
/ nested payloads / no-id fallback / no artist).
- 2 new tests in ``test_automation.py`` exercise the per-album
split end-to-end through ``process_wishlist_automatically``:
multi-album batch → two sub-batches each with album context;
mixed orphan + real album → one bundle batch + one residual.
1099 tests across wishlist + imports + downloads + automation +
playlist-sources + staging-provenance + track-number-repair
suites green. WHATS_NEW entry added under 2.6.3.
Now when an auto-wishlist scan finds 12 missing tracks from
Ryoto's "Cha-La Head-Cha-La", it runs ONE slskd / Prowlarr
album-bundle search for the release instead of 12 per-track
searches.
Root cause (#700): the Soulseek album-bundle path downloads whole
releases into a private staging dir, then per-track workers claim
those files via the staging-match shortcut. When slskd files arrived
without ID3 tags (common for FLAC rips), the staging cache fell back
to the filename stem as the title — and stems shaped like
"Artist - Album - 03 - Title" could not clear the 0.80 title-
similarity threshold against the clean Spotify track name. Every
track in the album went not_found, the batch ended "failed" in the
Downloads UI with an empty queue, and the bundle-downloaded files
just sat unused in staging.
Fix: in _staging_title_variants, add a trailing-title variant by
extracting the segments after a bare track-number block (e.g. "03")
between " - " delimiters. Conservative — only fires when a clear
digit segment is present, so real song titles with dashes like
"Hold Me - Live" are left intact. Generated as an additional variant
alongside the existing raw/compacted/feat-stripped/bonus-stripped
forms, so behavior on already-matching files is unchanged.
Downstream (#698): the album-bundle staging miss pushed every failed
track to the wishlist labelled as a playlist track, and a couple of
fallback paths in ensure_wishlist_track_format and the slskd-result
reconstruction hardcoded album_type='single' / total_tracks=1 on the
stored album dict. On wishlist requeue the path builder saw
album_type='single' and routed the download through single_path,
dumping the file in the Singles tree even though it belonged to an
album. (Running Reorganize would fix it because the DB album linkage
was still correct, but the file landed in the wrong place first.)
Fixes:
- new resolve_wishlist_source_type_for_batch() returns 'album' for
is_album_download batches; wishlist_failed.py now calls it instead
of hardcoding 'playlist'
- build_wishlist_source_context() threads album_context /
artist_context / is_album_download from the batch into the wishlist
row so future requeue logic has authoritative routing data
- the non-dict-album fallback in ensure_wishlist_track_format and
the slskd-result reconstruction default album_type='album' (and
total_tracks=0 = unknown) instead of lying with 'single'/1; the
existing setdefault chain handles dict-shaped album data unchanged
Tests:
- 2 staging-match tests pin the new tail-extraction behavior against
a realistic untagged slskd stem, plus a negative test that confirms
a dash-in-title without a digit segment still does NOT extract a
variant
- 2 payload tests pin the album_type='album' default for both
fallback paths
- 4 processing tests pin resolve_wishlist_source_type_for_batch()
and the album-context threading in build_wishlist_source_context()
3974 pass; no behavioural change on already-working flows.
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded.
Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path.
These files had silent `except Exception: pass` blocks but no module
logger. Added `import logging` + `logger = logging.getLogger(__name__)`
at the top of each, then replaced the silent excepts with
`logger.debug(...)`.
- core/replaygain.py — 4 sites (id3 txxx + vorbis + mp4 atom reads)
- core/wishlist/presence.py — 3 sites (wishlist row parsing + queries)
- core/runtime_state.py — 1 site (activity toast emit)
- core/automation/signals.py — 1 site (collect known signals)
- core/download_engine/rate_limit.py — 1 site (plugin rate_limit_policy)
- api/system.py — 1 site (hydrabase status probe)
- api/search.py — 1 site (hydrabase search)
Refs #369
- normalize album.total_tracks before comparing it in wishlist classification
- avoid mixed-type comparisons when provider payloads serialize track counts as strings
- add regression coverage for numeric strings and invalid values
- Let the wishlist service accept both track_data and spotify_track_data
- Preserve the backward-compatible wrapper while avoiding the keyword argument crash
- Add a regression test for the alias path
- add neutral wishlist payload helpers while keeping legacy Spotify aliases
- route wishlist removal and classification through generic track data
- keep API and service compatibility for existing callers
Both the auto and manual wishlist download paths called
`remove_tracks_already_in_library` before submitting the batch — a
serial DB lookup per track per artist (~1s/track on a 24-track
wishlist). The batches set `force_download_all=True` which is
explicitly documented as "skip the expensive library check" — the
pre-flight cleanup was contradicting that flag.
Removed the cleanup call from both flows. Kept `remove_wishlist_duplicates`
(fast SQL DELETE) and the standalone `/api/wishlist/cleanup` endpoint
that exposes the library scan as explicit user-triggered maintenance.
Safety check on the trade-off:
- post-processing at `core/imports/pipeline.py:576-624` already handles
re-downloads defensively: existing file with metadata → skip overwrite
+ delete source duplicate, no library corruption.
- Master worker's analysis loop normally removes wishlist entries for
found tracks via `_check_and_remove_track_from_wishlist_by_metadata`,
so stale wishlist entries should be rare in practice.
- Worst case for the rare orphan: one redundant download attempt that
the post-processing layer no-ops on. Bandwidth waste, not data damage.
Tests updated:
- `..._does_not_run_library_cleanup` (renamed from `_skips_enhance_tracks_during_cleanup`)
asserts no DB track-existence checks happen and no wishlist removals
fire — both `enhance` and "owned" tracks reach the master worker.
- `..._marks_batch_complete_when_wishlist_genuinely_empty` (renamed from
`..._after_cleanup`) covers the path where the wishlist starts empty.
Full suite: 1232 passing. Ruff clean.
The manual wishlist download endpoint blocked the request thread on a
slow library-cleanup pass before submitting the batch — for a 24-track
wishlist that's ~50 per-track DB lookups serialised in the request
handler, taking 30+ seconds before the frontend got a response. The
modal sat at "Pending..." with no progress visible the whole time.
Split start_manual_wishlist_download_batch into:
1. SYNC path (request handler):
- Generate batch_id, create download_batches entry with phase=analysis
and analysis_total=0 placeholder.
- Submit a single bg job (`_prepare_and_run_manual_wishlist_batch`) to
the missing-download executor.
- Return 200 with batch_id immediately. Frontend can start polling
/api/active-processes status right away.
2. BG path (executor thread):
- db.remove_wishlist_duplicates (slow-ish, single SQL)
- remove_tracks_already_in_library (the slow one — per-track DB checks)
- wishlist_service.get_wishlist_tracks_for_download
- sanitize + dedupe + filter (track_ids / category)
- Update batch.analysis_total with the real filtered count
- add_activity_item("Wishlist Download Started", ...)
- run_full_missing_tracks_process (master worker)
Edge case: if cleanup empties the wishlist, the bg job marks the batch
phase='complete' with error='No tracks in wishlist' (instead of the old
synchronous 400 response). Frontend status poll picks this up and the
modal can close cleanly.
Tests: existing 2 manual-download tests updated to drive the bg job
explicitly via a new `_run_submitted_bg_job` helper. Added 2 new tests:
- `..._returns_immediately_with_placeholder` — proves the sync path
doesn't trigger any cleanup or master-worker calls; analysis_total=0.
- `..._marks_batch_complete_when_wishlist_empty_after_cleanup` —
cleanup empties the list, master worker never invoked, batch ends
with phase='complete'.
Full suite: 1232 passing (was 1230). Ruff clean.
- remove the redundant wishlist-service injection from the runtime wrappers
- keep the package owning its own singleton service access
- simplify the route runtime API and update the wishlist tests to match
- add module-level loggers for the wishlist package instead of threading the web server logger through runtime objects
- default wishlist helper runtimes and cleanup helpers to their package logger while still allowing test overrides
- keep web_server.py as a thin caller that no longer injects its logger into wishlist flows
- extract the remaining wishlist endpoint behavior from web_server.py into core/wishlist/routes.py
- keep web_server.py as a thin Flask adapter around the new route helpers
- add tests that cover wishlist counts, stats, track listing, clear/remove flows, cycle updates, and album-track adds
- add core/wishlist as the home for wishlist payload, resolution, state, processing, reporting, and selection helpers
- move wishlist-specific tests into tests/wishlist alongside the new package layout
- keep web_server.py and the import/search callers as thin adapters for now
Broque Thomas
Antti Kettunen