Add MusicBrainz watchlist artist ID storage, badges, linked-provider editing, and per-artist preferred source support.
Backfill watchlist MusicBrainz matches from already-enriched library artists so existing MusicBrainz worker matches appear in watchlist cards and settings.
Extend bulk watchlist add, liked artist matching, artist map source picking, and service status labels to recognize MusicBrainz, with regression tests for watchlist ID persistence and backfill.
Register MusicBrainz as a first-class metadata source alongside Deezer, iTunes, Spotify, Discogs, and Hydrabase. Expose the shared client through metadata services, add the settings option, and expand the MusicBrainz search adapter with source-compatible artist, album, track, and detail methods.
Carry MusicBrainz IDs through similar-artist discovery, recommended artists, artist map serialization, and personalized playlist selection. Update DB migrations and lookup filters so similar_artist_musicbrainz_id is preserved on older schemas and used for source requirements and library exclusion.
Normalize MusicBrainz album adapter output for import context and add regression coverage for registry mapping, typed album conversion, and similar-artist filtering. Verified by user with 120 focused tests passing.
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded.
Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path.
Ensure the Amazon enrichment worker verifies its required columns before querying pending work or progress, preventing upgraded installs from spamming no-such-column errors when amazon_match_status is missing.
Add regression coverage for legacy databases without Amazon enrichment columns.
Preserve source metadata for seasonal and cached discover album modals so artist links use real provider IDs instead of falling back to library/name routes.
Treat source-only artist detail discographies as clickable missing releases and skip library-only ownership/enhancement checks.
Artist detail pages previously always pushed /artist-detail to the URL,
so refreshing the page or sharing a link would drop users on a broken
empty page with no artist loaded.
URL format is now /artist-detail/:source/:id (e.g.
/artist-detail/spotify/4tZwfgrHOc3mvqsCAfo4LT or
/artist-detail/library/42). The source segment lets the backend
synthesize a response from the right metadata client without a DB hit.
Changes:
Client routing (legacy shell + TanStack bridge)
- buildArtistDetailPath / _getDeepLinkArtistDetail added to init.js;
parse both new :source/:id and legacy bare :id formats so old
bookmarks still work
- navigateToPage passes artistId + artistSource through to the router
bridge, which builds the dynamic href instead of hardcoding route.path
- resolveShellPageFromPath / resolveLegacyShellPageFromPath use a prefix
match so /artist-detail/* resolves to artist-detail page-id
- globals.d.ts typed for artistId / artistSource options
- activateLegacyPath and syncActivePageFromLocation (popstate) both
restore artist from URL using skipRouteChange:true to avoid a
re-navigation loop back to /artist-detail
- loadInitialData restores artist from URL on page load (router not yet
mounted at DOMContentLoaded so legacy path runs unconditionally)
- Same-artist guard in navigateToArtistDetail prevents double-fetch
when the router fires activateLegacyPath after the initial navigation
Server
- artist_source_detail.build_source_only_artist_detail now resolves
artist name from the source API when none is supplied, so deep-link
restores with an empty name string still render correctly
Tests
- test_spa_deep_linking: /artist-detail/42 and /artist-detail/spotify/ID
both serve index.html
- bridge.test.ts: source-aware URL building and library fallback
- route-manifest.test.ts: prefix path resolution
- artist_source_detail: name resolved from source when input is empty
Add service-level coverage for the Enhanced Library I Have This flow: copying an existing source file, writing the target album DB row, preserving source audio, inheriting album identity tags, and migrating older track tables that lack disc_number.
Move the existing-file missing-track import workflow out of web_server.py and into core/library/missing_track_import.py.
Keep the Flask route focused on request wiring and response formatting while the service handles staging copy, post-processing, album identity tag inheritance, DB upsert, and media-server sync.
Add a conservative Soulseek album preflight scorer so album downloads choose a coherent slskd folder before per-track enqueue. The scorer compares album title, artist, year, track count, tracklist coverage, peer quality, and penalizes unexpected deluxe/remix/live-style folders.
Preserve hybrid source priority by only running Soulseek album preflight when Soulseek is the selected source or first in the hybrid order. If Soulseek is only a fallback behind another source, the normal hybrid flow is left alone.
Reuse the richest wishlist album context across tracks in the same album group so release date, artwork, album type, and album artist stay consistent for path generation. Also preserve peer-quality tie breakers when attempting equal-confidence candidates.
Tests cover correct-folder selection over larger wrong editions, Soulseek primary vs fallback hybrid behavior, shared wishlist album context, and peer-quality candidate ordering.
Three ruff S110 violations replaced with logger.debug calls:
- amazon_client.py:527 duration backfill ASIN search
- amazon_client.py:679 album metadata fetch in _fetch_album_metas
- amazon_worker.py:401 artist image backfill from albums
- Artist cards, hero section, and enhanced view now show Amazon Music badges
when amazon_id is populated (AMAZON_LOGO_URL constant, orange #FF9900 brand)
- Enhanced view artist and album match status rows include amazon_match_status
chip with click-to-rematch via openManualMatchModal
- getServiceUrl: added amazon (album/track ASIN → music.amazon.com) and fixed
missing discogs entries; serviceLabels adds tidal/qobuz/amazon
- Enhanced view enhanced-artist-id-badges includes amazon_id entry
- DB SELECTs for library artists list and artist detail now return amazon_id;
both response dicts include the field
- watchlist_artists migration adds amazon_artist_id column
- Watchlist config GET: amazon_artist_id in SELECT/WHERE/response (index 18)
- Watchlist artists list response includes amazon_artist_id
- link-provider endpoint: amazon added to valid_providers and col_map
- _populateLinkedProviderSection: amazonId param + Amazon Music source row
- Watchlist card source badges render Amazon pill (watchlist-source-amazon CSS)
- _openSourceSearch labels map includes amazon
- service_search: amazon_worker injected via init(); _search_service amazon branch
uses search_artists/albums/tracks, same {id,name,image,extra} return shape
- _SERVICE_ID_COLUMNS: amazon → amazon_id for artist/album/track
- _init_service_search call passes amazon_worker_obj
- amazon_client._fetch_album_metas: 5-minute TTL cache per ASIN — cached hits
skip _rate_limit() and HTTP call entirely; fixes ~10s artist detail load
- registry.py: removed amazon from METADATA_SOURCE_PRIORITY and
METADATA_SOURCE_LABELS — T2Tunes has no discography API, cannot serve as a
primary metadata source; Amazon remains a download source + ASIN enricher
- Settings metadata source dropdown and help text updated accordingly
The cap caused albums beyond position 10 to load without art on the
artist detail discography. T2Tunes search_raw naturally returns ~20
results per query, so album_candidates is already bounded — no explicit
cap needed.
Two bugs in the library artist detail page when Amazon is the source:
1. No album art: get_artist_albums returned Album dataclasses with
image_url=None — it collected ASINs but never called _fetch_album_metas.
Now fetches metas for up to 10 albums (same cap as search_albums),
populating image_url, release_date, and total_tracks on each Album.
2. No singles: Album.from_search_hit hardcodes album_type="album" and
T2Tunes exposes no release type in search results. Added inference:
total_tracks==1 → album_type="single", which routes them to the
singles bucket in the discography categorizer.
Also passes album_name through _strip_edition and artist through
_primary_artist in get_artist_albums (parity with search_albums).
3. amazon_id missing from artist_source_ids in get_artist_detail:
the discography lookup never received the stored Amazon slug so
it always fell back to name search. Added 'amazon': artist_info.
get('amazon_id') to the dict alongside spotify/deezer/itunes/etc.
Background worker matching library artists/albums/tracks to Amazon ASINs
via T2Tunes search. Follows same 6-tier priority queue as Deezer/iTunes/
Spotify/Qobuz/Tidal workers. Backfills artist thumbnails from album cover
stand-ins (T2Tunes exposes no direct artist images).
- core/amazon_worker.py: new AmazonWorker class with full parity
- database/music_database.py: expand _add_amazon_columns to cover
amazon_id/amazon_match_status/amazon_last_attempted on artists,
albums, and tracks (was artists-only)
- web_server.py: import, init, register in enrichment panel, add to
scan pause/resume dicts and rate monitor key map
- helper.js: WHATS_NEW 2.5.3 entry for enrichment worker
Schema: ALTER TABLE artists ADD COLUMN amazon_id TEXT with index, added via
_add_amazon_columns migration called after Discogs in _run_migrations.
SOURCE_ID_FIELD: add "amazon" -> "amazon_id" entry. find_library_artist_for_
source now looks up Amazon artists by slug before falling back to name match,
same as every other source. artist_source_detail already stamps artist_info
[source_id_field] = artist_id so the amazon_id is set on source-only payloads.
Tests: add "amazon": "amazon_id" to EXPECTED_SOURCE_ID_FIELD; revert test
assertion back to strict equality (SOURCE_ONLY_ARTIST_SOURCES == SOURCE_ID_
FIELD.keys() holds again now that amazon has a column).
Library upgrade: find_library_artist_for_source returned None immediately for
Amazon because SOURCE_ID_FIELD has no 'amazon' entry (no DB column for Amazon
artist IDs). The name-based fallback was unreachable. Fix: only skip the column
query when column is None, not the whole function — name lookup now runs for
any source when artist_name + active_server are provided.
Artist images: add AmazonClient._get_artist_image_from_albums so the standard
_get_artist_image_from_source path in metadata/artist_image.py can call it as
a fallback (same hook iTunes/Deezer/Discogs expose). Searches by unslugified
artist name, matches primary artist, fetches album cover from album_metadata.
Test: updated test_source_only_set_matches_mapping_keys → _contains_all_mapped_
sources to assert subset (not equality) — SOURCE_ONLY_ARTIST_SOURCES intentionally
includes sources without a DB column that rely on name-only lookup.
T2Tunes albumList entries may not include a release_date field, leaving the
$year path template empty. get_album() now falls back to the first track's
release_date (populated from the FLAC date tag via get_album_tracks) when
album metadata has none. Also try camelCase releaseDate key at all albumList
read sites (Album.from_metadata, get_album, _fetch_album_metas consumers).
1 new test: release_date backfilled from stream date tag when absent from
album metadata. date tag "2024-11-22" added to MEDIA_RESPONSE_FLAC fixture.
media_from_asin returns no duration data. get_album_tracks now does one
search_raw call using the album name + primary artist from stream tags,
filters hits by albumAsin == requested asin, and builds a duration_map
(track asin → duration_ms). Search failures are swallowed — duration_ms
falls back to 0 so the existing behaviour is preserved on error.
2 new tests: duration populated when search returns matching hit; duration
stays 0 when search endpoint returns an error.
release_date: T2Tunes album metadata may use camelCase releaseDate — try both
keys at all read sites (get_album, get_track_details, Album.from_metadata,
_fetch_album_metas consumers). Final fallback: s.date from stream tags, which
T2Tunes always populates from embedded FLAC/MP4 date tag. Wire s.date into
get_album_tracks items and get_track_details album.release_date so the $year
path template resolves correctly.
disc_number crash: .get('disc_number', 1) returns None when key is present but
value is None (Amazon stream info has Optional[int] for disc_number). Switch all
max() call sites and disc_num assignments to `or 1` guard:
- master.py: run_full_missing_tracks_process max() and disc_num read
- candidates.py: track_info and detailed_track disc_number reads
- web_server.py: enhanced and standard album download max() calls
AcoustID verification was quarantining every Amazon track because T2Tunes
embeds [Explicit] and [feat. X] in stream tag titles/artists, but AcoustID
returns bare titles — triggering version-mismatch rejection on every track.
- get_track_details: apply _strip_edition to name/album, _primary_artist to
artist; wire s.track_number / s.disc_number instead of hardcoded None
- get_album_tracks: apply _strip_edition to name, _primary_artist to artist
Also fix TypeError crash in album download paths when disc_number is None
(present in dict but explicitly None, so .get('disc_number', 1) returns None):
- master.py run_full_missing_tracks_process: or 1 guard on both max() and disc_num
- candidates.py track_info extraction: or 1 guard on both disc_number reads
- web_server.py enhanced + standard album download max() calls: or 1 guard
- All search_raw calls switched from single-type to types="track,album" — T2Tunes only
returns results when both types are requested together
- _fetch_album_metas: parallel fetch (up to 5 workers) of album cover art via
album_metadata(asin) — T2Tunes search results carry no image URLs
- search_tracks: populates image_url, release_date, total_tracks from album meta
- search_artists: strips feat. credits via _primary_artist() so "Artist feat. X" and
"Artist ft. Y" collapse to one "Artist" entry; uses album cover as artist image
stand-in (same approach as iTunes — T2Tunes has no artist images)
- search_albums: name-based dedup (display_name + artist key) instead of ASIN-based;
populates image_url, release_date, total_tracks from album meta (cap 10 ASIN fetches)
- _strip_edition(): strips [Explicit]/(Explicit) from track/album names — explicit is
the default version; Clean/Edited/Censored labels kept as-is so they stay distinct
- get_album(): applies _strip_edition to name and _primary_artist to artist so
MusicBrainz preflight matching doesn't fail on "[Explicit]" album names
- get_album_tracks(): populates track_number and disc_number from T2TunesStreamInfo
instead of hardcoding None — fixes track ordering in multi-track album downloads
- get_artist() / get_artist_albums(): _unslugify() converts slug artist IDs back to
search names; _primary_artist() in comparison handles feat-annotated results
- SOURCE_ONLY_ARTIST_SOURCES: added "amazon" so artist detail page doesn't 404
- build_source_only_artist_detail: added amazon_client param + dispatch branch
- web_server.py: resolve amazon_client in _build_source_only_artist_detail wrapper;
add source_override=="amazon" branch in get_spotify_album_tracks endpoint
- 77 tests covering all above paths; all pass
- Add 'amazon' to VALID_SOURCES (and transitively VALID_STREAM_SOURCES)
in core/search/orchestrator.py so the backend accepts it as a
requested source without returning 400
- Add resolve_client('amazon') case — mirrors musicbrainz pattern,
gets the cached AmazonClient from the metadata registry
- Add 'amazon' to _alternate_sources() so it appears as a tab when
another source is primary (always available, no credentials)
- Add SERVICE_CONFIG_REGISTRY entry 'amazon': {'always': True} so
/api/settings/config-status reports it as configured
- Add SOURCE_LABELS['amazon'] and SOURCE_ORDER entry in
shared-helpers.js so both enhanced search and global search show
the Amazon Music tab
- Add 'amazon' to _ALWAYS_CONFIGURED_SOURCES so the picker never
dims the tab (no credentials required)
- Add .enh-tab-amazon.active CSS (Amazon orange #FF9900)
- 3530 tests pass
Wires AmazonClient into the metadata source registry following the
exact same pattern as DeezerClient. No existing source paths touched.
- Add get_album_metadata / get_artist_info / get_artist_albums_list
aliases to AmazonClient (mirrors DeezerClient interface aliases)
- Register amazon in METADATA_SOURCE_PRIORITY and METADATA_SOURCE_LABELS
- Add _get_amazon_factory() + get_amazon_client() to registry.py
- Add amazon branch to get_client_for_source(); thread amazon_client_factory
kwarg through get_primary_client() and get_primary_source_status()
- Re-export get_amazon_client from the core.metadata_service shim
- Add Amazon Music option to Settings metadata source dropdown
- 3530 tests pass
The download monitor blocks post-processing with a bytes-incomplete guard:
if size > 0 and transferred < size: continue
_stream_to_file throttles engine updates to every 0.5s. The last tick before
the file finishes typically leaves transferred slightly below the Content-Length
size in the engine record. Other streaming clients (YouTube, Tidal, HiFi, etc.)
use their own download threads and don't track bytes at all, so size stays 0
and the guard is always skipped. Amazon was the only client hitting it.
Fix: just before returning the file path from _download_sync, write a final
engine record update setting size == transferred == out_path.stat().st_size
(the decrypted output size). The bytes-incomplete guard then sees
transferred == size and falls through to trigger post-processing normally.
`get_all_downloads` was calling `engine.get_all_records()` — a method that
doesn't exist on DownloadEngine. Same story for `cancel_record` and
`clear_completed`. The engine exposes `iter_records_for_source`, `get_record`,
`update_record`, and `remove_record` — matching what every other streaming
client (Deezer, HiFi, Qobuz, SoundCloud, Tidal, YouTube) already uses.
With `get_all_downloads` silently returning `[]` on every call (the missing
method raised, `except Exception: return []` swallowed it), the download monitor
never saw Amazon records as complete — tasks stayed stuck at 0% even after the
file had fully downloaded.
Changes:
- `get_all_downloads` → `iter_records_for_source('amazon')`
- `get_download_status` → `get_record('amazon', id)`, no try/except
- `cancel_download` → `get_record` check + `update_record` (Cancelled) +
optional `remove_record` — same pattern as deezer/hifi/etc
- `clear_all_completed_downloads` → iterate + `remove_record` for terminal
states; returns True on no-engine (nothing to clear = success)
- `_record_to_status` drops the `download_id` argument; reads `rec['id']`
instead (worker stores `'id'` in every record — `iter_records_for_source`
returns the full record dict)
Tests updated to match: `iter_records_for_source` mock replaces
`get_all_records`, cancel test verifies `update_record`+`remove_record`,
clear test verifies only terminal-state records are removed, graceful-error
test replaced with no-records boundary test (exception propagation is handled
at the engine aggregator layer, not per-plugin).
The engine worker stores the encoded filename under the key 'filename'
(see worker.py dispatch). _record_to_status was reading 'original_filename',
which always returns "" — so every DownloadStatus emitted by
get_all_downloads/get_download_status had an empty filename string.
The download monitor builds lookup keys as
_make_context_key(download.username, download.filename). With filename=""
the key was always "amazon::" which never matched the task's
"amazon::B0B1234||Artist - Title" key. Monitor never detected Amazon
download completions, so tasks sat stuck at Downloading 0% forever even
though the files had actually downloaded.
Also fixes tests that had the same wrong key.
AmazonDownloadClient was missing set_engine() and set_shutdown_check().
The download engine auto-wires plugins by calling set_engine(self) at
registration time if the method exists (engine.py:136). Without it,
_engine stayed None forever, causing every download() call to raise
RuntimeError("_engine is not set") — silently failing and marking all
tracks not found.
All other streaming clients (Deezer, Qobuz, Tidal, HiFi, SoundCloud)
expose set_engine(); Amazon now matches the pattern.
Tests added: set_engine wires _engine, set_shutdown_check wires callback,
set_engine unblocks download dispatch (the exact live failure mode).
`validation.py` had amazon absent from `_streaming_sources`, causing
Amazon TrackResult objects (bitrate=None, size=0) to fall through to
the Soulseek P2P code path and get rejected by
`filter_results_by_quality_preference`. Every album track was marked
not found.
Fix: add 'amazon' to every streaming-source guard tuple/set that was
previously missing it:
- core/downloads/validation.py — primary bug fix (quality-filter bypass)
- core/downloads/status.py — _STREAMING_SOURCE_NAMES frozenset
- core/downloads/task_worker.py — hybrid fallback client map
- core/imports/side_effects.py — || filename→stream-id extraction
- web_server.py — is_streaming_source, transfer list display,
candidate source label, _try_source_reuse, _store_batch_source
- tests/test_download_plugin_conformance.py — registry count + parametrize
Also updates the 2.5.3 What's New entry to drop the stale
"not yet wired" disclaimer.
Follows the exact same standard as Tidal, Qobuz, HiFi, and Deezer.
registry.py — import + register AmazonDownloadClient as 'amazon'.
amazon_download_client.py — read amazon_download.quality / allow_fallback
from config on init; pass quality as preferred_codec to AmazonClient;
_download_sync codec waterfall respects allow_fallback flag.
download_orchestrator.py — reload_settings() updates preferred_codec +
allow_fallback on the live client after a settings save. 'amazon' added
to _streaming_sources so search_and_download_best routes it correctly.
api_call_tracker.py — 'amazon' registered in RATE_LIMITS (120/min),
SERVICE_LABELS, and SERVICE_ORDER so API call monitoring shows Amazon.
web_server.py — 'amazon_download' added to the settings service loop.
'amazon' added to serverless_sources (no slskd probe needed). Streaming
file-finder extended to handle amazon username + ||asin||title encoding
(extension-less fuzzy match, same as Tidal/Qobuz/HiFi). New endpoint:
GET /api/amazon/test-connection → checks T2Tunes proxy status.
webui/index.html — amazon-download-settings-container: quality dropdown
(flac/opus/eac3), allow-fallback checkbox, test-connection button.
webui/static/settings.js — 'Amazon Music' added to HYBRID_SOURCES,
_hybridSourceEnabled, allSources mode list, loadSettings(), saveSettings()
payload, updateDownloadSourceUI() show/hide + auto-test. New
testAmazonConnection() function.
core/amazon_client.py — T2Tunes-backed metadata client following the
DeezerClient/iTunesClient contract. Exposes search_tracks, search_artists,
search_albums, get_track_details, get_album, get_album_tracks, get_artist,
get_artist_albums, get_track_features. T2TunesStreamInfo dataclass captures
the hex decryption key returned by the proxy (CENC/AES-128). Handles the
"stremeable" API typo. 0.5 s rate-limit guard + api_call_tracker.
core/amazon_download_client.py — DownloadSourcePlugin backed by the above
client. Codec waterfall: FLAC → Opus → EAC3. Downloads the encrypted MP4
container, decrypts with ffmpeg -decryption_key, yields the native audio
file (.flac / .opus / .eac3). Not yet wired into the app source registry —
validated in isolation only; see tests/tools/.
tools/t2tunes_probe.py + tools/t2tunes_media_plan.py — standalone CLI tools
used for live API exploration during development.
tests/tools/test_amazon_client.py — 72 unit tests (all mocked).
tests/tools/test_amazon_download_client.py — 52 unit tests (all mocked).
124 tests pass.
Reproduced: selecting Fresh Tape (or any kind never generated before)
and running the pipeline silently skipped — UI showed
"No tracks in Fresh Tape — skipping sync" with no clue why.
Root cause: ensure_playlist auto-creates the playlist row on first
access with `track_count=0` and `last_generated_at=NULL`, but
`is_stale=0` by default (the column default — fresh rows aren't
"stale", they're "never generated"). Pipeline only refreshed when
`is_stale=True` OR `refresh_first=True`, so first-run rows fell
through both branches → read the empty snapshot → skip.
Fix: pipeline now also refreshes when `existing.last_generated_at is
None`. Same control flow, one extra condition:
if refresh_first OR is_stale OR last_generated_at is None:
refresh
else:
read existing snapshot
This is the right signal: "has the generator ever run for this row"
is exactly what `last_generated_at` tracks (the column is set in
`_persist_snapshot` after every successful refresh).
Stubs in test_handlers_personalized_pipeline.py updated to expose
`last_generated_at` on their SimpleNamespace returns so the new
attribute read doesn't AttributeError. Fresh stubs get a non-None
timestamp so they're treated as already-generated; the new test
`test_never_generated_snapshot_triggers_first_refresh` pins the
first-run-forces-refresh behavior with `last_generated_at=None`.
Reproduced on the personalized playlist pipeline: selecting Fresh Tape
(or any kind) and running the automation surfaced
"Working outside of application context" in the UI.
Root cause: `get_current_profile_id` reads Flask's `g.profile_id` and
only catches `AttributeError`. Outside a request — automation engine,
sync threads, watchlist scanner — `g` raises `RuntimeError` instead,
so the except misses and the handler dies.
Mirrored playlist pipeline never hit this because it hardcodes
profile_id=1 in its sync call. The personalized pipeline calls
`deps.get_current_profile_id()` from a background thread, which is
what tripped the bug. Fresh Tape's generator also resolves the
profile via the same function — same path, same crash.
Fix: broaden the except to `(AttributeError, RuntimeError)` in all
three copies of the helper (`web_server.py`, `core/artists/map.py`,
`core/discovery/hero.py`). All three now safely degrade to profile_id=1
(admin profile) when called outside a request context — matches the
existing intent that single-admin installs Just Work.
No test changes — the existing pipeline tests stub the helper, so
they never exercised the bug. The fix is in the layer above the
stubs.
Snapshots now track when their source data changes. Watchlist scan
emits stale flags on the playlists whose underlying pool just got
refreshed; the next pipeline run sees the flag and regenerates the
snapshot before syncing, so the server playlist never lags the source.
Schema:
- new `is_stale INTEGER NOT NULL DEFAULT 0` column on
`personalized_playlists`, plus an idempotent ADD COLUMN migration
in `ensure_personalized_schema` for installs created before this PR.
- `PlaylistRecord.is_stale: bool = False` exposed on the dataclass so
callers can branch on freshness without re-querying.
Manager:
- new `mark_kinds_stale(kinds, profile_id=None)` flips the flag in
bulk for a list of kinds (used by upstream data refreshers).
- `_persist_snapshot` clears `is_stale = 0` on successful refresh.
- SELECT statements + `_row_to_record` updated to read the column
(with tuple-form length guard for safety).
Pipeline:
- `_build_payloads_for_kinds` now branches: refresh_first=True OR
`existing.is_stale` -> refresh_playlist, else read existing
snapshot. So the auto-refresh kicks in without needing the user to
toggle the refresh-each-run option.
Watchlist scanner emits stale flags at three sites:
- after `update_discovery_pool_timestamp` -> marks pool-fed kinds
stale: hidden_gems, discovery_shuffle, popular_picks, time_machine,
genre_playlist, daily_mix.
- after release_radar `save_curated_playlist` -> marks `fresh_tape`.
- after discovery_weekly `save_curated_playlist` -> marks `archives`.
All three calls go through a module-level `_mark_personalized_kinds_stale`
helper that builds a PersonalizedPlaylistManager with `deps=None` (only
DB access is needed for the flag update — no generator dispatch). Each
call is wrapped in try/except so a flag failure can never abort the
scan itself.
Tests:
- new `TestStaleFlag` class in `test_personalized_manager.py` (6
tests): default-false, single-kind flip, multi-kind, profile
scoping, refresh-clears, empty-list noop.
- two new pipeline tests pin the auto-refresh dispatch:
`test_stale_snapshot_auto_refreshes_even_without_refresh_first`
and `test_non_stale_snapshot_skips_refresh`.
- existing stub-manager `SimpleNamespace` returns gained
`is_stale=False` so the new attribute read doesn't AttributeError.
Full suite: 3391 pass.
User-facing WHATS_NEW entry added under 2.5.2 (above the prior
pipeline auto-sync entry) describing the auto-refresh behavior.
The action was registered + the block declared, but the automation
builder's per-action config renderer didn't have a case for
`personalized_pipeline` so users only saw the bare card with the
generic delay-minutes input — no way to select which playlists to
sync. This commit adds the multi-select picker.
Backend:
- `core/personalized/api.list_kinds(manager=...)` now optionally
takes a manager and includes the resolved variant list per kind
(calls each spec's variant_resolver(deps) when present). Singleton
kinds get an empty `variants` list. Variant-bearing kinds
(time_machine / genre_playlist / daily_mix / seasonal_mix) get
their full enumerated set.
- `web_server.py` `/api/personalized/kinds` route now passes a built
manager so the variants list lands in the response.
Frontend:
- `webui/static/stats-automations.js` `_renderBlockConfigFields`
gains a `personalized_pipeline` branch that renders a scrollable
multi-select picker:
- Singletons (Hidden Gems, Discovery Shuffle, Popular Picks,
Fresh Tape, The Archives) = one checkbox row per kind
- Variant kinds = a section header + one checkbox row per variant
(e.g. Time Machine: 1960s/1970s/.../2020s; Seasonal: halloween/
christmas/valentines/summer/spring/autumn)
- Pre-checks rows that match the existing `kinds` config on edit
- New `_autoLoadPersonalizedKinds(slotKey)` fetches `/api/personalized/kinds`
(cached after first load), renders the picker DOM, and pre-checks
saved selections via `data-kind` / `data-variant` attributes on
the checkboxes.
- `_renderBuilderCanvas` calls the loader for any `cfg-*-kinds-picker`
it finds in the freshly-rendered slots.
- The save-time `_collectActionConfig` walks the picker's checked
inputs (matched by `data-kind` attribute) and emits
`{kinds: [{kind, variant?}, ...], refresh_first, skip_wishlist}`
in the same shape the handler expects.
Tests:
- `tests/automation/test_automation_blocks.py::_FIELD_TYPES` adds
'personalized_playlist_select' so the block-shape regression test
accepts the new field type. (Test was failing because it whitelists
every field type used across all blocks.)
- 189 automation + personalized API tests pass; full suite intact.
Follow-up to the personalized-playlists standardization PR. New
`personalized_pipeline` automation action syncs selected discover-
page playlists (Hidden Gems / Discovery Shuffle / Time Machine /
Genre / Daily Mix / Fresh Tape / The Archives / Seasonal Mix) to
the active media server + queues missing tracks for download.
Same pattern as the existing mirrored `playlist_pipeline` but two
phases instead of four — no REFRESH (no external source to re-pull)
and no DISCOVER (manager-backed snapshots are already metadata-
matched). Pipeline shape:
SNAPSHOT → SYNC → WISHLIST
Where SNAPSHOT either reads the persisted track list from
`PersonalizedPlaylistManager` (default) or refreshes it first when
`refresh_first=true` (cron use case: regenerate Hidden Gems nightly
and sync the fresh set).
Shared helper extraction:
PHASE 3 (SYNC loop) + PHASE 4 (WISHLIST tail) lifted out of mirrored
`playlist_pipeline` into `core/automation/handlers/_pipeline_shared.py`
as `run_sync_and_wishlist(deps, automation_id, playlists, sync_one_fn,
sync_id_for_fn, ...)`. Both pipelines call it. Mirrored injects
`auto_sync_playlist` as the per-playlist sync function; personalized
injects a thin wrapper that launches `_run_sync_task` directly with
a pre-built tracks_json. Same sync-state polling / progress emission
/ status counting / wishlist trigger logic — 0 duplication.
Files added:
- core/automation/handlers/_pipeline_shared.py
- core/automation/handlers/personalized_pipeline.py
- tests/automation/test_handlers_personalized_pipeline.py
Files changed:
- core/automation/handlers/playlist_pipeline.py: PHASE 3+4 replaced
with shared helper call (~100 lines deleted, 1 helper invocation
added; behavior identical).
- core/automation/deps.py: new `build_personalized_manager` field
(lazy builder so the pipeline gets a fresh PersonalizedPlaylistManager
per run).
- core/automation/handlers/__init__.py + registration.py: register
`personalized_pipeline` action with the shared `pipeline_running`
guard so it can't overlap mirrored.
- core/automation/blocks.py: new `personalized_pipeline` block
declaration with config_fields (kinds multi-select, refresh_first,
skip_wishlist).
- web_server.py: thread `_build_personalized_manager` into
AutomationDeps construction.
- All 5 automation test fixtures: `_build_deps` adds
`build_personalized_manager=lambda: None` stub.
- tests/automation/test_handler_registration.py:
EXPECTED_ACTION_NAMES + EXPECTED_GUARDED_ACTIONS gain
`personalized_pipeline`.
Trigger schema:
{
"_automation_id": "...",
"kinds": [
{"kind": "hidden_gems"},
{"kind": "time_machine", "variant": "1980s"},
{"kind": "seasonal_mix", "variant": "halloween"}
],
"refresh_first": false,
"skip_wishlist": false
}
Tests (14 new, 178 automation total):
- _track_to_sync_shape: basic shape, source ID fallback chain,
no-id returns empty string
- empty config / non-list kinds / empty kinds list all return
error + clear pipeline_running flag
- _build_payloads_for_kinds: skips invalid entries, skips kinds
with no tracks, refresh_first vs ensure dispatch, payload shape
+ sync_id format, manager exception swallowed continues
- _sync_personalized_playlist: launches background thread + returns
status='started'
- happy path: stubbed sync_states drives helper to completion, flag
cleaned up
Full suite: 3383 passed.
Note: the trigger UI block declares config_fields but the frontend
doesn't yet render the `personalized_playlist_select` multi-select
type — usable today via API; polished UI ships in a follow-up
frontend PR.
CI ruff check failed on the seasonal_mix tuple-row coercion path
where a `zip(columns, row)` call lacked an explicit `strict=`.
Set `strict=False` to preserve the original intent (tolerant if
the row shape ever drifts from the column tuple). The SELECT
always returns 8 columns so the lengths match in practice; using
strict=False just avoids a future raise if a generator drift
changes that.
Live happy path stays unchanged: rows from sqlite3.Row hit the
`hasattr(r, 'keys')` branch above and never reach the zip line.
The zip branch only runs for plain-tuple rows in tests.
Adds the first quality feature on top of the manager: when
`config.exclude_recent_days > 0`, the manager drops any track from
the generator's output whose primary id was served by this kind
for this profile in the last N days.
Lives at the manager layer, not in each generator, so:
- generators stay focused on selection logic
- staleness behavior stays uniform across every kind
- enabling/disabling per playlist is just a config patch
Implementation:
- New `PersonalizedPlaylistManager._apply_quality_filters` runs after
generator returns, before `_persist_snapshot`.
- Reads recent ids via existing `recent_track_ids` accessor.
- Tracks without a primary id pass through unchanged (nothing to
dedupe on -- happens for sourceless tracks during edge cases).
- Returns a new list (never mutates input).
Default `exclude_recent_days = 0` preserves pre-overhaul behavior.
Per-playlist override via `PUT /api/personalized/playlist/<kind>/config`
with `{"exclude_recent_days": N}`. Recommended values:
- Discovery Shuffle: 1-3 days (high churn desired)
- Hidden Gems: 7-14 days (avoid same gems weekly)
- Time Machine / Genre: 30+ days (slow rotation OK, stable view preferred)
4 new boundary tests:
- Zero days = no filter (default behavior preserved)
- Positive days drops tracks served in window
- Filter preserves new tracks alongside dropped ones
- Tracks without primary id pass through unchanged
3369 tests pass total.
Note: listening-history cross-ref + seeded shuffle are deferred to
a future PR. Each requires deeper integration -- listening history
needs a play-events table the discovery pool can query against;
seeded shuffle needs the legacy generators to accept a seed param
without breaking their existing diversity / popularity logic.
Wraps the manager + generator dispatch behind one HTTP surface so
the UI can drop the patchwork `/api/discover/personalized/*` calls
in favor of a single REST shape. Legacy endpoints stay alive for
backward compat during the UI migration window.
New endpoints:
- GET /api/personalized/kinds — list every registered kind + metadata
- GET /api/personalized/playlists — list every persisted playlist for the active profile
- GET /api/personalized/playlist/<kind> — fetch singleton + tracks
- GET /api/personalized/playlist/<kind>/<variant> — fetch variant + tracks
- POST /api/personalized/playlist/<kind>/refresh — regenerate singleton
- POST /api/personalized/playlist/<kind>/<variant>/refresh — regenerate variant
- PUT /api/personalized/playlist/<kind>/config — patch singleton config
- PUT /api/personalized/playlist/<kind>/<variant>/config — patch variant config
Per-call manager construction wires the deps each generator needs:
- database (MusicDatabase singleton)
- service (PersonalizedPlaylistsService for legacy generator calls)
- seasonal_service (SeasonalDiscoveryService for seasonal_mix)
- get_current_profile_id (active profile accessor)
- get_active_discovery_source (source dispatcher)
API handlers themselves live as pure functions in
`core/personalized/api.py` so they're testable without Flask. The
Flask layer in `web_server.py` is a thin parse-body / call-handler /
jsonify wrapper.
11 new boundary tests (122 personalized total):
- list_kinds enumerates registry, exposes default config + tags
- list_playlists returns empty list when none exist, serializes
PlaylistRecord shape correctly
- get_playlist_with_tracks auto-creates on first access, returns
persisted tracks, raises ValueError on unknown kind
- refresh_playlist runs generator and returns track snapshot,
forwards config_overrides to the generator
- update_config patches stored config
3365 tests pass total. Manager construction triggers generator
registration via `from core.personalized import generators` import
side-effect.
Begins the standardization of the personalized-playlist subsystem.
Pre-existing state was a patchwork: Group A (Fresh Tape / Archives /
Seasonal Mix) lived in `discovery_curated_playlists` and
`curated_seasonal_playlists` with inconsistent shapes; Group B
(Hidden Gems / Discovery Shuffle / Time Machine / Popular Picks /
Genre / Daily Mixes) was computed on-demand by
`PersonalizedPlaylistsService` with no persistence -- every call
reran the generator with `ORDER BY RANDOM()` so results rotated.
Post-overhaul (this PR) every personalized playlist lands in one
unified storage layer with stable identity, persistent track lists,
explicit refresh, and per-playlist user-tweakable config.
Foundation in this commit (no behavior change yet):
- `database/personalized_schema.py`: 3 tables created idempotently
at app startup (wired into `MusicDatabase._initialize_database`).
- `personalized_playlists`: one row per (profile, kind, variant)
with config_json, track_count, last_generated_at,
last_synced_at, last_generation_source, last_generation_error.
Variant '' (empty string) for singletons; non-empty for
time_machine / seasonal_mix / genre_playlist / daily_mix.
- `personalized_playlist_tracks`: current snapshot per playlist.
Atomically replaced on refresh.
- `personalized_track_history`: append-only log powering the
`exclude_recent_days` config knob.
- `core/personalized/types.py`: `Track`, `PlaylistConfig`,
`PlaylistRecord` dataclasses. `PlaylistConfig.merged()` for
partial-update PATCH semantics; `Track.from_dict()` accepts the
legacy generator output shape unchanged.
- `core/personalized/specs.py`: `PlaylistKindSpec` (kind,
name_template, default_config, generator, variant_resolver) and a
module-level registry. Generators register at import time;
manager dispatches by kind.
- `core/personalized/manager.py`: `PersonalizedPlaylistManager` --
the only thing that touches the new tables. Owns:
- ensure_playlist (auto-create row from kind defaults)
- get_playlist / list_playlists
- refresh_playlist (atomic snapshot replace; generator exception
preserves previous good snapshot + records error on row)
- get_playlist_tracks
- update_config (deep-merge with stored config, including extra dict)
- recent_track_ids (staleness lookup for generators)
35 boundary tests in `tests/test_personalized_manager.py` pin every
shape: config round-trip / merge semantics / extra deep-merge /
defaults; Track.from_dict tolerance + primary_id fallback chain;
registry dedup / display_name with+without variant; manager
ensure_playlist auto-create + idempotency, variant separation,
required-variant enforcement, unknown-kind error; refresh persists
+ replaces atomically + survives generator exception with previous
snapshot intact + records source from first track + round-trips
nested track_data_json; update_config patch semantics; list_playlists
profile scoping; staleness history scoped to (profile, kind, days).
3304 tests pass total. Generators ship in subsequent commits on this
branch -- each kind migrated one at a time with its own per-kind
boundary tests.
Cleans up the four remaining inline callbacks at the bottom of
`web_server._register_automation_handlers` so the function is now
purely deps-construction + register_all + a logger.info line.
Lifted:
- `_progress_init`, `_progress_finish`, `_record_automation_history`,
and `_on_library_scan_completed` -> core/automation/handlers/progress_callbacks.py
Each is a top-level function that takes deps as a parameter; the
engine sees thin lambdas through `register_progress_callbacks` /
`register_library_scan_completed_emitter` (called from `register_all`).
Two new deps fields:
- `init_automation_progress` (delegates into the live progress tracker)
- `record_progress_history` (delegates into _auto_progress.record_history)
12 new boundary tests in tests/automation/test_progress_callbacks.py
pin every shape:
- progress_init forwards to init_automation_progress
- progress_finish skips when handler manages its own progress
(prevents double-emit of finished status)
- progress_finish: completed -> finished/Complete/success;
error -> error/Error/error; msg falls through error -> reason ->
status -> 'done'
- record_history threads the live db into the recorder
- on_library_scan_completed: no engine = noop, server type taken
from web_scan_manager._current_server_type, defaults to 'unknown'
- register_library_scan_completed_emitter: no scan manager = noop,
registered callback emits the right event when invoked
3256 tests pass, no regression.
Final state of `_register_automation_handlers`:
- Was: 1530 lines, 21 nested closures + 4 progress callbacks
- Now: ~50 lines, builds AutomationDeps and calls register_all
web_server.py: 34,220 -> 34,187 lines (-33 net, -1,406 across the
whole branch).
Final commit of the automation-handler refactor. With this commit
every closure that used to live in
`web_server._register_automation_handlers` is now a top-level
function in `core/automation/handlers/`.
Handlers extracted in this commit:
- start_database_update + deep_scan_library
-> core/automation/handlers/database_update.py
Both share the db_update_state monitoring pattern (poll until
status flips, stall detection emits warning at 10 min, 2-hour
outer timeout). Lifted into a shared `_run_with_progress` helper
inside the module so the per-handler bodies stay tiny.
- run_duplicate_cleaner -> core/automation/handlers/duplicate_cleaner.py
- start_quality_scan -> core/automation/handlers/quality_scanner.py
- clear_quarantine, cleanup_wishlist, update_discovery_pool,
backup_database, refresh_beatport_cache
-> core/automation/handlers/maintenance.py
Grouped because each body is short (~20-50 lines) and they share
no state — splitting into per-handler files would just add import
noise.
- clean_search_history, clean_completed_downloads, full_cleanup
-> core/automation/handlers/download_cleanup.py
Grouped because all three reach the download orchestrator,
tasks_lock, and download_batches/download_tasks accessors. The
full_cleanup multi-step orchestration shares phase-detection
logic with clean_completed_downloads.
- run_script -> core/automation/handlers/run_script.py
- search_and_download -> core/automation/handlers/search_and_download.py
`AutomationDeps` grew with the new dependency surface:
- get_db_update_state + db_update_lock + db_update_executor +
run_db_update_task + run_deep_scan_task
- get_duplicate_cleaner_state + duplicate_cleaner_lock +
duplicate_cleaner_executor + run_duplicate_cleaner
- get_quality_scanner_state + quality_scanner_lock +
quality_scanner_executor + run_quality_scanner
- download_orchestrator + run_async + tasks_lock +
get_download_batches + get_download_tasks +
sweep_empty_download_directories + get_staging_path
- docker_resolve_path + get_current_profile_id +
get_watchlist_scanner + get_app + get_beatport_data_cache
- set_db_update_automation_id (writes the legacy global so the live
DB-update progress callbacks still living in web_server.py keep
emitting against the active automation card)
`web_server._register_automation_handlers` is now ~50 lines: build
deps once, call register_all. The 667-line block of remaining
closure definitions and engine register calls is gone.
The final orphan was the `_db_update_automation_id` module global —
the DB-update progress callbacks at line ~14080 still read it
directly, so the extracted database_update handler propagates the
automation id through `deps.set_db_update_automation_id` (a closure
in web_server that writes the global). When the legacy callbacks
get extracted in a future PR the setter goes away.
Tests:
- tests/automation/test_handlers_maintenance.py adds 21 boundary
tests covering every newly-extracted handler shape: guard
short-circuits (already-running returns skipped), deps wiring
(set_db_update_automation_id called with the right id),
exception swallow contract, status returns, path-traversal
blocked in run_script, source-mode skip in clean_search_history,
active-batch skip in clean_completed_downloads, etc.
- 3244 tests pass (was 3223 — 21 new), no regression.
web_server.py: 35,593 -> 34,220 lines (-1,373 net across 3 commits).
Issue #1 from the extraction punch list is now COMPLETE.
Continues the lift from `web_server._register_automation_handlers`.
This commit extracts the four playlist-lifecycle closures:
- `refresh_mirrored` -> core/automation/handlers/refresh_mirrored.py
- `sync_playlist` -> core/automation/handlers/sync_playlist.py
- `discover_playlist` -> core/automation/handlers/discover_playlist.py
- `playlist_pipeline` -> core/automation/handlers/playlist_pipeline.py
The pipeline composes refresh + sync + discover, so all four ship
together. The pipeline imports the other three handler modules
directly (cross-handler call) instead of going through the engine,
preserving the "single trigger from the user's perspective" UX.
`AutomationDeps` grew to cover the new dependency surface:
- run_playlist_discovery_worker, run_sync_task, load_sync_status_file
(pre-existing background-task entry points)
- get_deezer_client, parse_youtube_playlist (per-source clients)
- get_sync_states (live mutable accessor for the sync UI's state dict)
`web_server._register_automation_handlers` now wires those plus the
existing infrastructure into a single `AutomationDeps` and calls
`register_all`. The 669-line block of closure definitions and engine
register calls (lines 959-1627 pre-edit) is gone -- the file shed
743 lines net on this commit.
`tests/automation/test_handlers_playlist.py` adds 17 new boundary
tests:
- discover_playlist: no_id error, specific_id starts worker, all=True
enumerates, no playlists in db
- refresh_mirrored: error path, source filter (file/beatport excluded),
Spotify happy path with auto-discovered marker, per-playlist
exception captured into errors counter
- sync_playlist: no_id, not_found, no_tracks, no-discovered-tracks
skip, discovered-track happy path, unchanged-since-last-sync skip
- playlist_pipeline: no_playlist clears running flag, no-refreshable
clears running flag, exception clears running flag
3223 tests pass. web_server.py: 35,593 -> 34,850 lines (743 removed).
Broque Thomas