Two related bugs from the Slipknot album never finishing.
1) _poll_album_bundle_downloads hung when the peer stalled. The finish check
needs every transfer terminal (completed/failed); the #715 grace only covers
'slskd says Completed but file not on disk'. A transfer stuck InProgress /
Queued, or dropped by slskd, is none of those — so it blocked both the finish
AND the grace exit, and the poll spun to the full ~6h timeout.
Add a bundle-level stall guard: track a progress marker (#terminal transfers,
total bytes across pending). If NOTHING advances for _stall_grace (180s) —
no terminal transition AND no pending byte movement — the peer has stalled;
mark the stuck transfers failed so the existing finish/all-failed checks
resolve the bundle with whatever completed (missing tracks then fall back to
the per-track matcher). Conservative: only trips when EVERYTHING is frozen,
so a slow-but-progressing or still-queued transfer is unaffected.
2) Failed batches lingered in the UI forever ('No tracks loaded'). The
auto-cleanup gate removed only complete/error/cancelled phases — 'failed'
(e.g. an album-bundle hard failure) was missing, so it never aged out. Add
'failed' to the terminal set so it's removed after 5 minutes like the others.
Tests (tests/test_soulseek_album_poll_stall.py): stalled peer → gives up with the
completed subset (not the deadline); progressing bundle not falsely stalled;
all-stalled → empty; dropped transfers stall out; clean finish unaffected.
124 download/soulseek tests pass; ruff clean.
make_wishlist_batch_row / _run_wishlist_cycle annotate params with Optional,
but the typing import only had Any/Callable/Dict. Slipped past py_compile +
tests because 'from __future__ import annotations' makes annotations strings
(never evaluated at runtime), but ruff flags it statically (F821).
Stage 2: the manual 'Download Wishlist' flow now calls the same
_run_wishlist_cycle engine the auto timer uses, so a manual scan runs the exact
same code path as an auto scan. The old bespoke manual orchestration (build
payloads + SERIAL inline dispatch) is deleted — its grouping/dispatch was a
near-duplicate of auto's that had already drifted.
Behavior changes (all intended, discussed):
- Manual now dispatches album bundles in PARALLEL (album pool) like auto, instead
of serially on one thread. A single cycle='albums' engine call covers the whole
selection (albums bundled, singles/ungroupable -> per-track residual), so no
'both cycles' pass is needed.
- The manual placeholder batch_id is reused as the engine's first sub-batch
(first_batch_id), so the modal's existing poll target stays valid.
- WishlistManualDownloadRuntime gains album_bundle_executor (wired in web_server,
falls back to the shared pool when unset).
- 'Don't start manual while auto is running' is unchanged — the existing route
guard (is_wishlist_actually_processing -> 409) already covers it; no queue added.
NOT touched: process_wishlist_automatically's behavior (proven by test_automation
staying green in Stage 1) and the per-track download mechanics.
test_manual_download.py rewritten to characterize the new behavior (engine
dispatch via the executor, parallel, placeholder reuse, album-context). Full
wishlist suite green (131); wishlist + automation = 392 passed.
Stage 1 of unifying the auto + manual wishlist flows. Extract the
group -> per-album+residual batches -> register -> dispatch logic that lived
inline in process_wishlist_automatically into a standalone _run_wishlist_cycle()
engine (built on make_wishlist_batch_row). The auto path now just calls it.
Per-flow differences are arguments (auto_initiated stamps the auto-only fields +
selects auto vs manual naming/logging; first_batch_id lets a caller reuse a
pre-created placeholder). Album batches dispatch to the dedicated album pool,
residual to the shared pool (unchanged from #740).
Auto behavior is PROVABLY unchanged: its full characterization suite
(test_automation.py) stays green (10/10), and the whole wishlist suite passes
(131). This commit does NOT touch the manual flow yet (Stage 2) and does not
change what auto does — it only moves auto's logic behind a shared entrypoint
the manual flow will call next.
The auto and manual wishlist flows each built the same ~20-field
download_batches row in separate places (auto album, auto residual, manual
placeholder, manual sub-batches) — four near-identical literals that could (and
did) drift apart, producing subtly different batch shapes between the flows.
Extract make_wishlist_batch_row() as the single source of truth: it emits the
consistent core field set, with the genuinely per-flow differences as explicit
arguments — initial phase ('queued' for auto / 'analysis' for manual), the
auto-only auto_initiated/auto_processing_timestamp/current_cycle via
extra_fields, and album-vs-residual contexts. All four sites now go through it,
so every wishlist batch has an IDENTICAL shape (this also removes the field
drift that confused the modal-hydration code).
Deliberately NOT unified — and left explicit in each caller, per the
'don't cargo-cult genuinely-different code' principle: the grouping decision
(auto groups only on the albums cycle), batch-id allocation (manual reuses the
caller's placeholder id for the first sub-batch), and dispatch (auto
parallel-submits album batches to the dedicated pool + residual to the shared
pool; manual runs them serially on one thread). Those are real behavioral
differences, not duplication.
Behavior-preserving: verified safe to normalize the row shape (grep confirmed
every reader uses .get() with defaults, no key-presence checks). The existing
auto (test_automation.py) and manual (test_manual_download.py) characterization
suites stay green = differential proof of identical behavior. Adds
test_batch_factory.py (core fields, album/residual, extra_fields, no shared
mutable state, consistent key shape). 131 wishlist tests pass.
A 2.6.3 change (c3b88e69) split the wishlist albums cycle into one batch per
album. Each album batch runs an INLINE-BLOCKING soulseek/torrent/usenet
album-bundle download (album_bundle_dispatch.try_dispatch ->
download_album_to_staging) that holds its worker thread for the whole
search+download. All of these were submitted to the shared 3-worker
missing_download_executor -- the same pool used for per-track downloads AND the
manual 'Download Wishlist' analysis.
So a large Album-Completeness 'Fix all' (-> ~819 wishlist tracks -> ~82 per-album
batches) saturated all 3 workers with blocking album downloads; the manual
wishlist analysis could never get a thread ('Library Analysis' stuck on
Pending), the other ~79 batches sat in phase='queued' forever, and auto-cleanup
(which only evicts terminal-phase batches) never cleared them -> jam until
restart. Fixing batch STATUS would not help: the threads are blocked inside the
download, not waiting on a phase flip.
Fix: add a dedicated bounded album_bundle_executor (max_workers=3) and route the
AUTO per-album bundle batches to it, keeping the shared pool free for analysis /
per-track / the manual wishlist (which always starts now). Hung/slow album
downloads can only delay other album downloads, never the user-facing path.
Additive and decoupled; the submit site falls back to the shared pool when the
album pool isn't wired (older callers / tests) so behavior is unchanged there.
The manual path is untouched (it already runs album bundles serially on a single
thread, by design).
Tests (tests/wishlist/test_automation.py): album sub-batches route to the
dedicated pool while the residual per-track batch stays on the shared pool;
fallback-to-shared-pool when no album pool is wired. Existing auto-processing
tests still green (fallback preserves prior behavior). 707 passed across
wishlist + downloads suites.
The same provider ID is stored under inconsistent column names across tables
(deezer_id vs deezer_artist_id vs album_deezer_id vs similar_artist_deezer_id;
spotify/itunes keep an entity qualifier, others don't; musicbrainz uses three
nouns), so code checks 2-5 name variants everywhere.
Add core/source_ids.py as the single source of truth for (provider, entity) ->
column, with accessors that read an ID from a dict/sqlite3.Row robustly
(canonical column first, then known aliases). NO database columns are renamed —
these are the real names today; the registry just centralizes the knowledge.
Targeted adoption (behavior-identical, verified):
- core/artist_source_lookup.SOURCE_ID_FIELD now derives from the registry
instead of duplicating the mapping (values unchanged).
- web_server.py artist-detail builds artist_source_ids via source_id_map(...)
instead of a hand-rolled per-source .get() dict.
Broader call-site adoption deferred as clearly-scoped follow-up.
Tests: tests/test_source_ids_registry.py (canonical columns, alias fallback,
canonical-preferred, sqlite3.Row, source_id_map, SOURCE_ID_FIELD unchanged).
Existing artist_source_lookup + artist_full_detail suites still green.
Migration state was scattered across PRAGMA-table_info guards, sentinel marker
tables (_genius_search_fix_applied, ...) and metadata-flag rows
(id_columns_migrated, ...), with no single source of truth and no schema
version — so a half-migrated DB was undetectable.
Add a non-gating backstop: a schema_migrations(name, applied_at) ledger plus a
_sync_migration_ledger pass (runs last in init) that back-fills the ledger from
the existing signals and stamps PRAGMA user_version. ADDITIVE only — existing
migrations keep their own idempotency gates; nothing decides whether a
migration runs based on the ledger or the version. New one-time migrations call
_record_migration (the genres migration already does).
Tests: tests/test_db_migration_ledger.py — table exists, user_version stamped,
record idempotent, genres recorded on fresh init, backfill from flag + marker,
absent signals not recorded.
artists.genres / albums.genres stored EITHER a JSON array (new writes) OR a
legacy comma-separated string (old writes), forcing every reader to
try-JSON-then-split. Add a marker-gated one-time migration
(_normalize_genres_to_json) that rewrites legacy rows to JSON in place,
mirroring the readers' exact parse (JSON list, else comma-split/strip/
drop-empties) so genre VALUES are unchanged — only the storage format.
Per-row diffed (already-canonical rows untouched, no churn) and non-fatal on
error, consistent with the other migrations. Readers still tolerate both
formats, so this breaks nothing; it just removes the dual-format debt.
Tests: tests/test_db_genres_json_normalization.py — CSV->JSON, JSON-unchanged,
whitespace/empties dropped, albums table, legacy-reader-equivalence,
idempotent re-run, marker set on fresh init.
amazon_artist_id is added to watchlist_artists via ALTER (music_database.py
~1732), but both table-rebuild migrations — the spotify_id-nullable fix
(_fix_watchlist_spotify_id_nullable, two CREATE variants) and the
profile-scoped UNIQUE rebuild — recreated the table from a hardcoded column
list that omitted amazon_artist_id. Because shared_cols filters new_cols
against the old table, the column and any stored Amazon artist IDs were
silently dropped on every init (fresh OR upgraded), so Amazon watchlist IDs
never persisted at all.
Fix: add amazon_artist_id to all three rebuild CREATE schemas, both rebuild
new_cols lists, and the base CREATE TABLE (so fresh installs are consistent
and don't rely on the ALTER). Purely additive, column-named inserts + Row
factory mean column position is irrelevant.
Tests (tests/test_db_watchlist_amazon_id_migration.py): drive the real
migrations via MusicDatabase() against a seeded pre-migration temp DB and
assert the column + data survive; differential-proven to FAIL pre-fix.
Continue the design-system unification (kettui UI-consistency item):
migrate the five remaining compact button families onto the shared
.btn .btn--sm primitive + color modifiers, and drop their bespoke base
CSS (net -125 lines of CSS).
- ya-header-btn (Your Albums/Artists, discover.js-injected) -> .btn .btn--sm
.btn--secondary; ya-refresh/ya-settings/ya-viewall co-modifiers kept.
- explorer-action-btn (Playlist Explorer) -> .btn--secondary / .btn--primary.
- repair-bulk-btn -> .btn--secondary / .btn--primary / .btn--warning (fix-all).
- enhanced-bulk-btn (Library bulk bar, library.js-injected) -> .btn--primary/
--secondary/--danger; class kept as a hook for the mobile.css size
override + the .tag-write / .rg-analyze special colors.
- profile-create-btn (init.js-injected) -> .btn .btn--block .btn--primary;
class kept for the scoped .profile-edit-buttons flex:1 rule.
mini-nav-btn deliberately left as a distinct icon-button archetype.
Formalize the compact 'toolbar' button tier as design-system modifiers
(.btn--sm), plus a full-width (.btn--block) and amber caution
(.btn--warning) modifier, so the many smaller per-page buttons can share
the .btn primitive without being forced to the large default size.
First adopter: the Sync page header buttons (.sync-history-btn) now use
.btn .btn--sm .btn--secondary. The class is kept as a JS/onboarding
selector hook; .auto-sync-manager-btn still tints Auto-Sync accent.
The watchlist + wishlist header/overview buttons used a bespoke
.watchlist-action-btn family (different padding/radius/font and white
primary text) instead of the shared .btn design-system primitive.
Migrate all 11 of them to .btn / .btn--primary / .btn--secondary /
.btn--danger so they match the rest of the app, and drop the now-dead
CSS.
The .watchlist-batch-remove-btn / .wishlist-batch-remove-btn hook
classes are kept on the remove buttons (their !important red overrides
compose correctly over .btn--secondary). Static HTML only; no JS-injected
usages, and mobile.css overrides target .playlist-modal-btn, not these.
Reorder the sidebar nav so Downloads sits between Wishlist and
Automations. Mobile nav reuses the same .nav-button elements and the
helper/onboarding references are selector-keyed, so no other changes
are needed.
The Tools-page Database Updater dropdown only offered Incremental and
Full Refresh, even though the backend (/api/database/update with
deep_scan) and the dashboard Deep Scan button already supported a deep
scan. Wire the missing option into the Tools UI:
- Add a "Deep Scan" option to the #db-refresh-type dropdown.
- handleDbUpdateButtonClick now sends { deep_scan: true } for that
option (deep scan takes precedence server-side) and confirms first,
since deep scan removes stale entries — mirroring the dashboard flow.
Frontend-only; the progress/status handler already drives the bar from
the backend phase ("Deep scan: ...") and the help/docs copy already
described all three modes.
Search results live in an overlay dismissed by an outside-click handler
whose allow-list omitted the floating media player. Clicking the mini
player to open the now-playing modal (or clicking inside that modal)
registered as an outside click and tore the results down, forcing a
re-search.
Add the media player containers (#media-player mini bar and
#np-modal-overlay expanded modal) to the dismiss allow-lists in both the
Search page (search.js) and the global search widget (downloads.js),
which share the same outside-click pattern. Additive change: only adds
exceptions, so every existing dismiss case is unchanged.
Reported by CubeComming: importing media keeps the track artist correct
(e.g. Billie Eilish) but changes the album-artist tag ("Albuminterpret") to
"Unknown Artist", breaking grouping in the media server.
Cause: in extract_source_metadata (core/metadata/source.py), album_artist is
seeded from the resolved track artist, then overridden by the album CONTEXT's
first artist. When the album lookup comes back unresolved, that first artist is
the literal "Unknown Artist" placeholder — which is truthy, so it clobbered the
real artist.
Fix: treat "Unknown Artist" (and empty) as a non-value — only let the album
context override the album_artist when it names a real artist. A genuine album
artist (e.g. "Various Artists") still overrides as before.
Tests: tests/metadata/test_album_artist_unknown.py — placeholder doesn't
clobber, real album artist still used, no-album-context falls back to track
artist, empty doesn't clobber. (Pre-existing test_album_mbid_cache.py failures
are an unrelated sandbox DB disk-I/O issue.)
Reported by @Yug1900: a Spotify playlist's overview shows the correct count
(e.g. 203) but the sync modal lists only 100, and only 100 would sync.
Root cause in /api/spotify/playlist/<id> (the Spotify-tab track fetch): owned
playlists fetch + paginate fine (sp.next loops over all pages). But when the
authenticated playlist_items call FAILS (intermittently, often a 403 on
followed / not-owned playlists) it fell straight back to the public embed
scraper, which is hard-capped at ~100 tracks and has no pagination — and the
result was returned as if complete. The overview total is correct because it
comes from the playlist *metadata* listing, which succeeds independently.
Fixes (additive — the owned/working path is unchanged):
- Retry the items fetch once before resorting to the embed scraper, so a
transient failure no longer silently truncates a large playlist.
- Guard against silent truncation: compare fetched count against the
playlist's known total; if short (or via the capped embed fallback), log a
warning and return `incomplete: true` + `expected_total` instead of
presenting a partial list as complete.
No behavior change when the fetch succeeds in full (incomplete=false, extra
fields ignored by the frontend). Lets a future UI tweak surface
"got 100 of 203 — retry" rather than silently dropping tracks.
Low-risk tidy-up for the full-bleed "exception" pages that aren't carded.
Every page already gets a 40px gutter from .page, but the exception pages were
piling on inconsistent extra padding (library +20px, active-downloads +28/32px,
discover/docs +0) — giving accidental 60 / 68-72 / 40 effective gutters.
Drop the redundant container padding on library and active-downloads so the
single .page 40px gutter is the shared, intentional outer spacing across the
full-width exception pages. discover (centered max-width) and docs (sidebar
layout) keep their functional layout; library's mobile padding override is
unaffected.
Standardize the sync page's outer spacing to match the other pages. Like
settings, its .sync-header and .sync-content-area were siblings directly under
.page (no wrapper) — wrap both in a single .page-shell div so it becomes the
floating card with consistent margin/padding. HTML-only change.
Watch: .sync-content-area uses height:95% (grid) — fine against an auto-height
card, but to be confirmed visually (library's full-height grid was the one
that didn't fit a card).
Add the canonical .tab (bordered rounded-pill filter tab) and .card ("glass"
content card) primitives as the documented design-system standard for new
markup and the React pages — modeled on the cleanest existing looks
(watchlist filter pill; dashboard service/stat card).
Deliberately NOT migrating the existing tab/card components onto them: the
current implementations are visually divergent and JS-coupled (active-state
toggled by class name, cards built in JS), so a blind consolidation risks
subtle regressions. These primitives let new/React code be consistent now;
the legacy components migrate when visually verifiable / in React.
Unused classes -> zero visual change to the current UI.
Migrate the wishlist add-to-wishlist modal buttons onto the shared .btn
primitive: primary -> .btn--primary, secondary -> .btn--secondary, the green
download CTA -> new .btn--download modifier. Added a shared .btn.loading state
(amber pulse, reusing the existing pulse-loading keyframe) since
confirm-add-to-wishlist-btn toggles `loading` via JS (wishlist-tools.js).
Removed the dead .wishlist-modal-btn* rules and re-scoped the mobile
full-width override to `.wishlist-modal-actions .btn`.
Start of the button-consolidation pass (kettui's #1). The app had ~236 button
classes / ~8-10 distinct looks with heavy near-duplication.
Introduce a canonical .btn design-system primitive (base + .btn--primary /
.btn--secondary / .btn--danger), modeled on the dominant existing look
(accent-gradient primary, translucent ghost, semantic danger) and built on the
accent CSS vars. New markup and the React pages should use this; existing
per-page button classes will migrate onto it family by family.
First family migrated: the config/settings modal buttons (.config-modal-btn*,
4 static uses, no JS refs) -> .btn .btn--primary / .btn--secondary. Removed the
now-dead .config-modal-btn* rules and re-scoped its mobile full-width override
to `.config-modal-actions .btn`.
Visible change is minor by design (padding 28->24px, gradient direction
normalized). Proof step for sign-off on the .btn look before rolling wider.
Settings was the one flat page with no single wrapper — its .dashboard-header
and .settings-content sat as siblings directly under .page. Wrap both in a
single .page-shell div so the page becomes a floating card with the header
banner at the top, matching the dashboard structure. HTML-only change (no CSS:
.settings-content keeps its minor `0 4px` inner padding).
Library is intentionally NOT converted — its full-height artist grid + A-Z
jump rail overflow a margin:20px card, so it stays flat as a documented
exception (same category as search/discover/active-downloads).
Follow-up to the poll fix, covering the two things that blocked a
successful end-to-end album import once the poll itself stopped
freezing:
1. Staging dir permissions
The album-bundle private staging path defaults to
'storage/album_bundle_staging' -> /app/storage, but /app/storage was
never created or chowned by the image (unlike /app/Staging,
/app/Transfer, etc.), and /app is root-owned. The copy failed with
"[Errno 13] Permission denied: 'storage'" under the non-root soulsync
UID. Added /app/storage to the Dockerfile build-time mkdir+chown and
the entrypoint PUID/PGID chown, exactly like the sibling runtime dirs.
2. Client->local path resolution
Usenet/torrent clients report save paths from inside THEIR OWN
container (e.g. SAB '/data/downloads/music/<album>'); SoulSync often
mounts the same files at a different point ('/app/downloads/<album>').
Feeding the client path straight to the audio walker yields
"No audio files found" though the files are physically present.
New resolve_reported_save_path():
a. use the reported path as-is if readable (mirrored mounts),
b. apply explicit download_source.usenet_path_mappings
({from,to}, Sonarr/Radarr-style) for non-shared layouts,
c. basename fallback under SoulSync's own download roots —
zero-config for the standard shared-volume arr setup.
Wired into both call sites in usenet.py AND torrent.py
(download_album_to_staging + _finalize_download), logging any
translation and including the resolved path in the no-audio error.
Tests: resolver verbatim / explicit-mapping / basename-fallback /
priority / not-found / empty / mapping-miss-then-basename. ruff +
compileall + pytest green (645 in the download suites).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the playlist-explorer page from a flat padded container to the
.page-shell floating card. Drop its bespoke `padding: 24px 32px`; keep the
full-height flex layout (display:flex / column / min-height:100%) since the
explorer fills the viewport.
Visible change by design. Watch: the full-height min-height:100% inside a
margin:20px card may run slightly tall — to be confirmed visually.
First of the "flat -> card" conversions. The automations list view sat
directly on the page background (.automations-container = bare padding) while
its inner .dashboard-header is the same header dashboard uses. Adopt
.page-shell so the page becomes a floating gradient card structurally
identical to the dashboard (page-shell card > dashboard-header > content).
- Drop .automations-container's bespoke `padding: 20px 24px` (card padding now
from .page-shell); keep the class as the mobile/JS hook.
- Add `page-shell` to the container in markup.
Visible change by design (this page was not previously a card). Mobile keeps
its existing .automations-container padding override.
First step of the page-layout-shell standardization (kettui's UI-consistency
point #1). The dashboard, tools, watchlist and wishlist pages each defined a
byte-identical "card" container (padding 28px 24px 30px, margin 20px, gradient
bg, radius 24px, border + border-top, layered shadow) under four different
class names.
Extract that into a single `.page-shell` primitive (modeled on the canonical
dashboard/stats look) and have the four pages adopt it. Each keeps its bespoke
class for page-specific extras and as a JS/mobile hook:
- dashboard-container: keeps display:flex / column / gap:25px
- watchlist/wishlist-page-container: keep position:relative
- tools-page-container: no extras (box now fully from .page-shell)
Zero visual change: computed styles are identical (declarations relocated, not
altered), and mobile.css overrides still target the retained bespoke classes.
Per-page themed headers (watchlist amber, etc.) are intentionally NOT touched.
The class name is intended for reuse by the React pages too, so the primitive
is shared across both stacks.
Next (wave 2): migrate settings / automations / playlist-explorer / library
onto .page-shell, which snaps their slightly-off spacing to canonical.
The earlier #721 fix tolerated a ~10s "completed but no save_path"
window, but the real production stall sits upstream of that: SABnzbd
removes a finished download from the queue and runs par2 verify /
repair / unpack *in History*, exposing the live stage in the slot
`status` ('Verifying' / 'Repairing' / 'Extracting' / 'Moving' / ...)
with `storage` empty until the final move. `_parse_history_slot` mapped
EVERY non-'Failed' status to 'completed', so a still-extracting 1.7 GB
FLAC album looked "completed with no save_path" the instant download hit
100%. The poll burned its completed-no-path budget mid-PP and bailed,
freezing the UI on the last download emit (the stuck-at-99%/100%
signature). SAB then finished fine — which is why the job shows
Completed in History but SoulSync never staged it.
Root fix
- `_parse_history_slot` routes `status` through `_map_state`, so PP
stages stay NON-terminal: the poll keeps waiting (as 'downloading')
for as long as post-processing takes and only a real 'Completed'
flips to terminal success. `save_path` is trusted only on true
completion (mid-PP path fields may point at the incomplete dir).
Supporting / defensive
- `UsenetStatus.incomplete_path`: surfaced separately from save_path
(SAB `incomplete_path`) and used by the poll loops as a LAST RESORT
after the completed-no-path window, to recover the case where
`storage` never lands but the files are physically on disk.
- `poll_album_download`: dedicated, configurable completed-no-path
window (~120s via `download_source.album_bundle_completed_no_path_seconds`)
decoupled from the ~10s transient-miss window; incomplete_path
fallback; a 30s heartbeat log so the previously-silent poll loop is
diagnosable.
- `usenet.py` `_download_thread`: per-track parity — it was erroring
immediately on the first completed-no-path read.
- `album_bundle_dispatch.py` / `status.py` / `monitor.py`: use the
project `get_logger` so download-flow logs land in app.log under the
`soulsync.*` namespace (they were console-only before, which hid the
`[Album Bundle] flow failed` line during triage).
Tests
- PP-history state mapping; end-to-end Hunky Dory PP regression
(download -> Verifying/Extracting in History past both budgets ->
Completed+storage -> success); completed-no-path window +
incomplete_path fallback; per-track thread parity. ruff + compileall +
pytest all green (the only local failures are environmental: missing
tzdata + local tools/ffmpeg.exe, neither present on CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final cluster: the four structurally-identical snapshot endpoints
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles) ->
core.discovery.endpoints.save_bubble_snapshot(...), wired via
_save_source_bubble_snapshot. All four validate a payload key, persist via
db.save_bubble_snapshot(kind, items, profile_id=...), and return a count +
timestamp; they differ only by:
- payload_key ('downloads' for discover, 'bubbles' for the rest) + its
no_data_error message.
- snapshot_kind, success_noun, and the info/except log subject + noun
("downloads"/"artists"/"albums/tracks"/"charts").
get_database / get_current_profile_id injected; get_json (request.json) invoked
inside the try, preserving the original 400/500 behavior incl. traceback dump.
Tests: +5 (missing key 400, None body 400, happy path with kind/profile/count/
timestamp, discover_downloads variant, exception -> 500). Full discovery suite:
210 passed.
web_server.py: -98 lines.
Ninth cluster: update_<source>_playlist_phase for the five sources sharing the
identical validation + full-message response (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube) -> core.discovery.endpoints.update_playlist_phase(...),
wired via _update_source_playlist_phase + the _PHASE_LIST/_PHASE_LIST_YT
constants.
Per-source params:
- valid_phases — YouTube additionally allows 'parsed'.
- apply_extra_fields — Deezer/Qobuz/Spotify-Public also persist
download_process_id / converted_spotify_playlist_id from the body; Tidal and
YouTube do NOT, so they pass False (kept strictly 1:1 — the generic won't
apply those keys for them even if a caller sent them).
- not_found_message / error_label; get_json invoked inside the try.
NOT folded in: iTunes-Link — uses data.get('phase') (no "Phase not provided"
400) and returns a no-message payload.
Tests: +7 (404, missing-phase 400, invalid 400, happy path with extra-fields
suppressed, extra-fields applied when enabled, YouTube 'parsed' allowed,
exception -> 500). Full discovery suite: 205 passed.
web_server.py: -123 lines.
Eighth cluster, the heavyweights (~110 lines each). The fix-modal
update_<source>_discovery_match for the four sources with the identical
structure (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.update_discovery_match(...), wired via
_update_source_discovery_match. Applies the user-selected Spotify track to the
discovery result (status/artist/album/duration/spotify_data/match-count) and
writes the manual fix to the discovery cache.
Per-source pieces are params:
- source_log_label / error_label.
- original_track_key ('tidal_track' / 'deezer_track' / ...).
- original_artist_getter: Tidal handles string-or-object artists
(first_artist_str_or_obj); the rest assume strings (first_artist_plain).
- web_server helpers (join/extract artist, build_fix_modal_spotify_data,
cache-key, get_database, active-discovery-source) injected.
- get_json passed as a callable and invoked INSIDE the try, preserving the
original's "request.get_json() inside try" behavior (malformed body -> 500).
NOT folded in (genuinely divergent): iTunes-Link (saves spotify_data directly
via a different cache signature), YouTube (multi-key original_track fallback),
ListenBrainz (entirely different unmatch-capable structure, no cache write),
Beatport.
Tests: +9 (extractors; 400/404/400 guards; full happy path with result
mutation + duration formatting + match-count + cache-save args; no-increment
when already found; cache error swallowed; get_json raise -> 500). Full
discovery suite: 198 passed.
web_server.py: -400 lines.
Seventh cluster: start_<source>_sync for the five sources with the identical
flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube) ->
core.discovery.endpoints.start_sync(...), wired via _start_source_sync.
Validates phase, converts discovery results, seeds sync state, posts a
"... Sync Started" activity item, and submits to the sync executor. Per-source
pieces are params:
- sync_id_prefix (f"{prefix}_{key}"), not_found/not_ready messages, convert_fn.
- name/image accessors: Tidal reads an object (playlist_name_obj/
playlist_image_obj), the rest a dict (playlist_name_strict/playlist_image_dict).
- activity_label vs error_label DIFFER for Spotify-Public ("Spotify Link
Sync Started" activity, "Spotify Public" logs).
- submit_sync_task glue (_submit_sync_task) closes over sync_executor /
_run_sync_task / get_current_profile_id so the helper stays global-free.
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
task WITHOUT a playlist_image_url arg), Beatport (extra debug logging, chart).
Tests: +6 (404, not-ready 400, no-matches 400, full happy path with
state/sync-infra/submit/activity assertions, resync phases allowed,
exception -> 500). Full discovery suite: 189 passed.
web_server.py: -172 lines.
Sixth cluster: the bulk-hydration get_<source>_playlist_states endpoints for
the five sources that build the identical per-entry dict + {"states": [...]}
shape (Tidal, Deezer, Qobuz, Spotify-Public, iTunes-Link) ->
core.discovery.endpoints.get_playlist_states(states, *, error_label,
info_log_label=None), wired via _get_source_playlist_states.
iTunes-Link is the only one of the five without the "Returning N stored ..."
info log, so info_log_label is optional (iTunes passes None to suppress it).
NOT folded in: the YouTube/ListenBrainz get_all_*_playlists endpoints. They
return {"playlists": [...]} (different key) with a different field set
(url / created_at / playlist, no discovery_results) and filter out
mirrored_/profile-scoped entries — genuinely divergent, kept as-is.
Tests: +4 (list build + last_accessed bump + exact shape, empty, optional ids
default None, missing-required-field -> 500). Full discovery suite: 183 passed.
web_server.py: -116 lines.
Fifth cluster: reset_<source>_playlist for the four sources with byte-
identical bodies (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.reset_playlist(states, key, *, label,
not_found_message), wired via _reset_source_playlist. Resets phase/status to
'fresh', clears discovery/sync fields, cancels any discovery_future, and
preserves the original playlist payload.
Left with their own bodies (genuinely divergent):
- YouTube: status -> 'parsed' (not 'fresh'), no download_process_id, logs the
playlist name, "reset to fresh state".
- ListenBrainz: status -> 'cached', logs playlist title, returns
{"success": True, "phase": "fresh"} (different payload), _lb_state_key.
- iTunes-Link: state.update(...), no info log, "iTunes Link reset to fresh
phase".
Tests: +4 (404, full clear + playlist preserved + future cancelled, no-future
path, exception -> 500). Full discovery suite: 179 passed.
web_server.py: -100 lines.
Fourth cluster: get_<source>_discovery_status (all eight sources, Beatport
included) -> core.discovery.endpoints.get_discovery_status(states, key, *,
not_found_message, error_label), wired via _get_source_discovery_status.
Unlike sync-status, the discovery-status response shape is byte-identical
across every source (phase/status/progress/spotify_matches/spotify_total/
results/complete), so Beatport folds in here too. Only the 404 string
("... discovery not found" vs "... playlist not found" vs "Beatport chart
not found") and the except-log label vary. ListenBrainz key via _lb_state_key.
NOT touched this cluster: get_*_playlist_state (the sibling endpoints).
Those genuinely diverge per source — different id-key name (playlist_id /
url_hash / playlist_mbid), presence of url / created_at / download_process_id,
Tidal's playlist.__dict__ serialization, and YouTube's strict (non-.get)
field access. Folding them would need a flag pile that wouldn't be a clean
1:1, so they keep their own bodies.
Tests: +4 (404, full response + last_accessed bump, complete=False when not
'discovered', missing-field -> 500). Full discovery suite: 175 passed.
web_server.py: -155 lines.
Third cluster: the get_<source>_sync_status routes (Tidal, Deezer, Qobuz,
Spotify-Public, iTunes-Link, YouTube, ListenBrainz) -> core.discovery.
endpoints.get_sync_status(...), wired via _get_source_sync_status glue.
This cluster carries the real per-source quirks, all captured 1:1 as params:
- not_found_message (iTunes-Link uses "iTunes Link not found").
- error_label vs activity_subject — these DIFFER for Spotify-Public: the
activity feed says "Spotify Link playlist ..." while the except log says
"Error getting Spotify Public sync status".
- playlist-name accessor, three styles lifted verbatim as named helpers:
playlist_name_attr_or_unknown (Tidal: object .name), playlist_name_strict
(Deezer/Qobuz/Spotify-Public/iTunes: state['playlist']['name'], can raise),
playlist_name_safe (YouTube/ListenBrainz: .get default). The strict getter
preserves the original's behavior of raising -> 500 AFTER phase/sync_progress
were already mutated.
- ListenBrainz key via _lb_state_key (caller-resolved).
Beatport stays separate (different payload: status not sync_status, sync_id,
no lock, chart key).
Tests: +9 (3 name accessors incl. raise/fallback semantics; status 404s,
running-no-mutation, finished+activity, error+revert+activity, and strict-
getter-missing -> 500 after partial mutation). Full discovery suite: 171 passed.
web_server.py: -244 lines.
First cluster of the per-source playlist-discovery deduplication. The
convert_<source>_results_to_spotify_tracks functions (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube, ListenBrainz) plus the already-generic
_convert_link_results_to_spotify_tracks were byte-identical apart from the
source label used in their log line.
Lift the shared body into core/discovery/endpoints.py as
convert_results_to_spotify_tracks(results, source_label); the 7 web_server
functions become 1-line delegations (names/signatures unchanged, so all
callers and behavior are identical — 1:1).
Beatport is intentionally NOT folded in: its converter coerces artist
objects to strings and emits a different track shape (source field, album
dict), so it keeps its own implementation.
Tests: tests/discovery/test_discovery_endpoints.py (12) pin both input
shapes (manual spotify_data / auto spotify_track+found), optional
track/disc numbers, falsy-0 omission, field defaults, skip-on-neither,
order preservation, if/elif precedence, empty input.
web_server.py: -209 lines. Full discovery suite: 151 passed.
Follow-up to the album-art resolution fix. That change upgraded MusicBrainz
Cover Art Archive thumbnails (/front-250) to the bare /front original — but
/front redirects to archive.org, which is unreliable: probing release-group
covers showed intermittent HTTP 500s (same URL 500s one second, serves the
next) and multi-MB originals (2.9 MB seen). The result was the user-reported
flakiness: cover art that "sometimes works, sometimes shows nothing", and a
huge image embedded into every track when it did work.
The sized thumbnails (/front-250, -500, -1200) are served by CAA's own CDN,
not the archive.org redirect — which is why /front-250 (240p) was always
reliable. Upgrade to /front-1200 instead: 1200x1200 is a massive jump from
240p, reliably CDN-served, and a sane ~40 KB instead of multi-MB.
Applied in all three CAA spots for consistency: the _upgrade_art_url helper
(embed + cover.jpg paths) and both prefer_caa ("CCA") blocks, which fetched
the bare /front directly with no fallback — so CCA-on users hit the same
flakiness. _fetch_art_bytes still falls back to the original /front-250 if
/front-1200 is ever refused.
Tests updated to assert the 1200px target, idempotency, and that the bare
/front original is intentionally left untouched.
User report: embedded album art came out ~600x600 while the cover.jpg in
the folder was high-res. The cover.jpg path upgraded the source CDN URL
to its highest resolution, but the tag-embed path fetched the raw URL —
so iTunes art embedded at its 600x600 default, Spotify at 640, Deezer at
1000. The "Write Tags to File" retag path had the same gap (Deezer-only
upgrade), and MusicBrainz art was worse still: every Cover Art Archive
URL is built as the /front-250 thumbnail, so MB-sourced downloads
embedded 250x250.
Factor the resolution upgrade + fetch into two shared helpers in
core/metadata/artwork.py and route every art path through them:
_upgrade_art_url(url) — bump to the source's highest resolution:
- Spotify (i.scdn.co) -> original master (~2000px+)
- iTunes (mzstatic.com) -> 3000x3000
- Deezer (dzcdn) -> 1900x1900
- Cover Art Archive -> /front original (was /front-250)
_fetch_art_bytes(url) — upgrade, fetch, and fall back once to the
original size if the CDN refuses the larger one (non-regressive).
Now consistent across: embed-into-tags (post-process), folder cover.jpg
(post-process), and the enhanced-library "Write Tags to File" retag flow.
The YouTube path already upgraded via Album.from_spotify_album, unchanged.
De-duplicates the per-source upgrade code that was copied across sites
and drops the now-unused urllib import from tag_writer.
Not covered (follow-up): Last.fm / Amazon / Tidal / Qobuz have no
explicit upgrade yet — some already serve full-res, others may hand over
a capped size that passes through unchanged.
Tests: new tests/metadata/test_artwork_resolution.py pins every upgrade
(Spotify 300/640->master, iTunes 100/600->3000, Deezer->1900, CAA
thumbnail->original, unrecognized/empty unchanged) and the fetch
fallback. Updated the two tag_writer fallback tests to patch the network
at its new home in artwork.
The four artist-hero buttons (Artist Radio, Watchlist, Download
Discography, Enhance Quality) had drifted apart visually — different
sizes, weights, radii, hover treatments, and ad-hoc colors. Unify them
on the Download Discography look (the nicest of the set): accent-style
gradient fill, matching border, light-tinted text, compact 7x16 / 12px /
700 sizing, and a translateY(-1px) + colored glow on hover.
To keep them distinguishable, each carries its own hue within that
shared recipe:
- Artist Radio -> violet (#8b5cf6)
- Watchlist -> theme accent (keeps its amber "watching" state)
- Download Discography -> theme accent + shimmer (the primary action)
- Enhance Quality -> cyan (#4fc3f7, its original signature color)
Also:
- Drop the shimmer from the three secondary buttons — four simultaneous
shimmers were distracting; it now marks only the primary action.
- Remove the `margin: 12px 0 4px` on `.discog-download-wrap` (now
`margin: 0; display: inline-flex`) that pushed the discography button
~4px below its siblings in the centered flex row.
- Include Artist Radio in the mobile button sizing override (was missing).
BoulderBadgeDad
Tyler Richardson-LaPlume