When a grabbed release fails (transfer error / peer-cancel / never lands), the engine
now retries instead of giving up — the music-style depth:
- Grab stores the OTHER accepted results as a retry pool + the search context (schema
v16: candidates / search_ctx / tried_queries / tried_files / attempts).
- core/video/retry.py (pure, tested): plan_retry() → try the next-best candidate; when
the pool is dry, next_query() generates an ALTERNATE query (movie: drop the year; TV:
numbering variants) to re-search; budget MAX_ATTEMPTS=6. merge_candidates dedupes
against already-tried releases.
- Monitor: on failure, _fail_or_retry hops to the next candidate inline; if none, flips
the row to a new 'searching' state and a background requery thread re-searches the
alternate query, evaluates against the profile, and starts the best fresh hit — or
marks failed once truly exhausted. 'searching' rows are owned by their thread.
- Page: 'Searching' status (Trying another release…) + a 'Nx' attempt badge.
16 tests (retry engine + candidate-retry transition); ruff clean on touched files.
Grabs now carry the movie's identity so the Downloads cards are rich, not anonymous:
- video_downloads gains media_id / media_source / year / poster_url (schema v15 +
migration); grab stores them (passed from the get-modal → download view → grab).
- Cards show the POSTER in the art tile (emoji fallback), 'Title (Year)', a quality
chip (1080p · BluRay · X265), and an ↗ Open button that jumps to the movie/show
detail page (dispatches soulsync:video-open-detail). Cancel/retry unchanged.
16 tests green, ruff + balance clean.
Fourth service. Rides the YouTube-video enrich path (like RYD/SponsorBlock), keyless,
on by default → toggle in the YouTube-Extras frame.
- DeArrowWorker: fetches the branding API and stores the first non-original crowd
title for a cached YouTube video. apply_youtube_dearrow + youtube_video_dearrow_title
DB methods; dearrow_status added to the youtube_enrich next/breakdown whitelists.
- Schema: dearrow_title/status/attempted on youtube_video_stats (video_schema.sql +
_COLUMN_MIGRATIONS for existing DBs).
- config GET/POST dearrow_enabled; manager orb (blue 🏹, video-kind) + status poll;
YT video-detail panel shows a 'DeArrow' alt-title (payload via youtube.py + CSS).
- 4 new tests + fixed-set/config assertions.
30 backfill tests green. (My change is ruff-clean; the 12 pre-existing S110s in
api/video/youtube.py are unrelated tech debt on this branch, left untouched.)
Adds a VideoBackfillWorker base that enriches already-identified items BY id
(vs the matcher workers), with the exact same lifecycle + get_stats() shape so
the engine registry, /api/video/enrichment routes, and Manage-Workers modal
drive them identically.
Workers:
- fanart.tv (free key): gap-fills logo/clearart/banner/backdrop/poster art
- OpenSubtitles (free key): records subtitle-language availability per title
- Return YouTube Dislike (no key): like/dislike estimates on cached videos
- SponsorBlock (no key): crowd segments per video
DB: youtube_video_stats + youtube_video_segments tables; fanart/subs columns;
backfill_next/mark/breakdown + youtube_enrich_* helpers; likes/dislikes merged
into get_channel_videos. Seam tests in tests/test_video_backfill.py.
TV episode rows show runtime; channel video cards showed nothing per video. The
InnerTube lockup already carries both — now we capture duration ('12:34') + an
approximate view count (parsed from '2.6M views') in innertube_parse_video_items,
remember them on youtube_channel_videos (new duration/view_count cols + migration),
and render a duration badge bottom-right of the thumb + 'N views' in the card meta.
The background stream backfills them onto the recent (yt-dlp) videos too, so the
whole catalog gets them. Live-validated (MrBeast: 32:08 / 50M, 30/30). 65 tests green.
Channel pages re-fetched everything every open (re-stream + ~3s yt-dlp metadata).
Now we remember what we learn about a channel and serve it cache-first:
- schema: youtube_channel_videos (list) + youtube_channel_meta (avatar/subs/tags/
banner); dates stay in youtube_video_dates, merged on read. DB: cache_/get_
channel_videos + cache_/get_channel_meta (upserts COALESCE so refreshes never
drop fields).
- enricher: _enrich now REMEMBERS a channel — caches the full InnerTube catalog
(list, via new innertube_channel_catalog) + metadata + dates, not just dates.
Since it already sweeps followed channels, watchlisted channels get pre-warmed
in the background → opening them is instant.
- channel endpoint: cache-first. A remembered channel renders from cache with NO
network (returns from_cache:true); a miss resolves live (yt-dlp) and remembers
it. The /videos batch endpoint caches every page it streams.
- frontend: cache hit renders the full catalog instantly, then re-streams to
refresh QUIETLY (new uploads/date fixes); only the first (miss) load shows the
'loading full history' banner.
Live-validated (MrBeast catalog: 90/3 pages, all titled+dated). 190 tests green.
Channels enriched before InnerTube existed (partial coverage, e.g. CaseyNeistat
36, Good Mythical MORE 25) were locked behind the 24h coverage gate and would
never reach the full ~240-date catalog on their own. Tag each enrichment row
with the source ('innertube'|'fallback'); legacy rows (method NULL after the
additive migration) bypass the gate ONCE so they re-enrich and upgrade, then
settle under the normal window. Non-destructive: existing follows, wished videos
(24), and cached dates (1176) are untouched — only coverage refreshes on next
open/sweep. Migration is additive (method TEXT); regression test added.
Follow a channel → a background job fetches its FULL upload-date catalog so the
channel page's year-seasons populate fully, cached permanently (one-time per
channel, instant after).
- core/video/youtube.py: proxy_channel_dates() — no-key BULK dates via Piped/
Invidious instances (tries several, paginates); parse_proxy_dates handles both
shapes. The clean path the user wanted.
- core/video/youtube_enrichment.py: YoutubeDateEnricher daemon — enqueue(channel)
→ proxy bulk → cache; per-video yt-dlp only as a throttled fallback (cap 60,
0.4s) for wished videos when every proxy is down. Skips channels enriched in
the last 24h.
- db: youtube_channel_enrichment table + mark/check + wishlisted_video_ids_for_
channel. Enqueued from /youtube/follow and on opening a followed channel.
Scoped to FOLLOWED channels only (per request). 174 tests green; enricher imports
nothing from music.
Flat listing has no upload dates, so channels showed one 'All Videos' season.
Now real year-seasons, filled from cheap sources and cached so they grow:
- core: parse_rss_dates / channel_recent_dates — one public RSS GET dates the
~15 most-recent uploads (no yt-dlp, no bot risk).
- db: youtube_video_dates cache table + cache_video_dates / get_video_dates.
- /youtube/channel merges cached + RSS dates onto the flat videos (year-seasons)
and caches what it learns; /youtube/video caches each fetched date too — so
expanding/wishing videos progressively fills the catalog's years, instant on
repeat. Dateless tail groups as 'Earlier videos'. Missing-only toggle hidden
for channels. 84 db + 84 youtube/api tests green.
Full day-one coverage still needs the YouTube Data API's publishedAt (optional,
yt-dlp can't match it cheaply) — RSS + cache is the no-key best.
The two new partial indexes (idx_video_wishlist_video/_channel on source_id /
parent_source_id) were in video_schema.sql, which runs via executescript()
BEFORE the column ALTERs. On a fresh DB the CREATE TABLE includes the columns so
it's fine, but on an existing (pre-bridge) DB the table already exists without
them, so 'CREATE INDEX ... ON video_wishlist(source_id)' threw 'no such column'
and the whole init rolled back — the video side wouldn't load at all.
Move those two indexes out of the schema into VideoDatabase._ensure_indexes(),
run AFTER _ensure_columns(). Regression test simulates a pre-source DB and
asserts the in-place upgrade + the youtube path both work. 81 DB tests green.
Bridges YouTube onto the existing watchlist/wishlist tables (the chosen
approach) via a generic source/source_id (+ parent_source_id on wishlist) and a
stable surrogate for the NOT NULL tmdb_id, so existing dedup/group-by machinery
is untouched. SCHEMA_VERSION 13, additive column migrations.
- Channel follow = video_watchlist kind='channel' (source='youtube').
- Wished video = video_wishlist kind='video' grouped by parent channel.
- youtube_surrogate_id(), add/remove/list/hydrate channels, add_videos_to_wishlist,
query_youtube_wishlist (channel=group, videos=newest-first feed),
youtube_wishlist_counts, scoped removal.
- Kept wishlist_counts/watchlist_counts byte-identical (exact-equality tests);
YouTube counts live on their own method. 80 DB tests green.
More data in the roomier episode cards (asked for): episodes now carry a synopsis
(new video_wishlist.episode_overview, SCHEMA_VERSION 12 + migration; captured at
add-time, and the art-backfill fills it for old rows from the same tmdb_season
call). The card shows a 2-line synopsis under the meta line and is now clickable
-> opens the show detail (episodes have no page of their own).
Organize the wishlist two ways: added 'Oldest first' (FIFO) alongside 'Recently
added' (newest) — query_wishlist gains the 'oldest' sort for both movies + shows.
Tests: FIFO/newest ordering (with pinned add-times) + overview roundtrip/backfill.
105 passed. Music wishlist untouched.
Season tiles showed the show poster because the wishlist had no season art. Now
stored + used: new video_wishlist.season_poster_url (SCHEMA_VERSION 11 + migration);
the get-modal captures each season's poster at add-time (owned -> /poster/season
proxy, tmdb -> direct); query_wishlist exposes season.poster_url; tiles render the
real season poster (falling back to show poster, then placeholder).
Backfill generalized: /wishlist/backfill-stills -> /wishlist/backfill-art fills
BOTH stills and season posters from the same cached tmdb_season call (it already
returns the season poster). The page fires it once when either is missing.
Tests updated/added: art backfill targets + season-poster set + endpoint. 104 passed.
#4 acquisition progress: a thin done÷wanted bar across each bubble's bottom
(forward-prep — fills once the download engine lands).
Expanded-view richness: episodes now carry a still thumbnail. New video_wishlist
.still_url column (SCHEMA_VERSION 10 + migration); the get-modal captures the
still per episode (owned -> /poster/episode proxy, tmdb -> direct still_url) and
sends it through add; query_wishlist returns it; the episode row renders a 16:9
thumb (film-frame placeholder when absent) in a roomier expanded tile.
Subtle video identity: the bubbles are now rounded-SQUARES (the music orbs stay
circles). Every rule stays scoped under .vwsh-nebula — verified no bare .wl-*
rules in the video CSS, so the music wishlist is untouched.
Tests: +1 (still roundtrip). Backend 102 passed.
The curated 'get this' list. Atomic units are movies and episodes; adding a
whole show/season expands into episode rows (show/season are bulk ops). Upcoming
episodes stay out — the watchlist/calendar promote them once they air.
- video_wishlist table + two partial unique indexes (one per movie tmdb_id, one
per (show tmdb_id, season, episode)) so the shapes don't collide and re-adds
upsert. SCHEMA_VERSION 8 -> 9 (executescript creates it on existing DBs).
- DB: add_movie_to_wishlist, add_episodes_to_wishlist (bulk), remove_from_wishlist
(movie/show/season/episode scope), query_wishlist (movies | shows grouped
show->season->episode w/ wanted/done roll-ups, searched+paged), wishlist_counts,
wishlist_state (hydration).
Tests: +7 (idempotent upserts, show-tree grouping, scoped removes, movie/episode
same-tmdb don't collide, hydration, search+paging). DB suite: 68 passed.
Owning a still-running show means you want its new episodes, so it's on the
watchlist without a click. Implemented as a computed default + explicit-override
so it stays correct:
- video_watchlist gains a 'state' column: 'follow' (explicit) | 'mute' (a
tombstone — user un-followed an airing show that's on by default, so the
default must not silently re-add it).
- Effective watchlist (list/state/counts) = explicit follows ∪ library shows
whose status isn't ended/canceled, minus mutes. Computed at READ time, so it
always tracks the library + a show's status — no scanner hook, no re-seeding.
- remove() now writes a mute tombstone (idempotent) instead of deleting; add()
sets state='follow' and clears any mute. Scoped to the active video server.
The existing library-card eye now paints 'watched' on airing shows by default;
clicking mutes, clicking again re-follows.
4 tests updated/added incl. the airing-default + mute + re-follow flow. 80 video
tests green.
A curated follow-list for the video side, mirroring the music watchlist. v1 is
membership only — the monitoring/discovery engine is a later phase.
Backend:
- video_watchlist table (kind 'show'|'person', keyed on tmdb_id — the stable
cross-context id both carry; library_id kept when owned). NOT the existing
shows.monitored flag (that defaults to 1 / is library-only / has no people).
- VideoDatabase: add/remove/list/state/counts (upsert COALESCEs library_id +
poster so a TMDB re-add can't wipe known data).
- /api/video/watchlist {GET, /add, /remove, /check, /counts}.
- query_library now selects s.tmdb_id so show cards can carry the key.
Frontend:
- video-watchlist-btn.js: shared eye button (the music ya-watchlist-btn mirror)
— build/toggle/hydrate, one delegated capture-phase click handler, broadcasts
soulsync:video-watchlist-changed so pages can react.
- Watchlist page (new subpage + video-watchlist.js): Shows / People tab switcher,
poster grid to detail-page quality, reloads each visit, drops cards on unfollow.
- Wired the eye onto library TV-show cards (movies excluded — wishlist, not
watch) + hydrate on render.
Tests: 6 new (DB upsert/COALESCE/state/counts + endpoint roundtrip/validation).
76 video tests green. Other card surfaces (cast, search, similar, filmography)
are the same VideoWatchlist.btn(...) one-liner — wired next.
A new isolated Calendar page (/api/video/calendar) — every upcoming episode for
your owned shows across a real 7-day week (today first), as art cards sorted by
air time with a per-cell breathing colour glow.
- Air times: enrich shows with TVDB airsTime (new shows.airs_time column +
migration); cells show + sort by time, streaming (untimed/00:00) = "Anytime".
One-time background backfill re-queues already-matched shows for the time.
- Click an episode → styled modal (show backdrop hero, episode still/synopsis,
air date+time, owned badge, genres, "about the show"), with an explicit
"Open full show page" action instead of navigating on click.
- Isolated: reads only video_library.db, writes nothing to the music side.
OMDb now has the same setup as TMDB/TVDB: a yellow dashboard orb (★ glyph) that
spins/idles in the worker-orb animation, an entry in Manage Workers (Ratings
coverage cards, pause/resume, retry, search), and a BACKGROUND ratings pass.
- Worker 'ratings mode' (is_ratings): instead of a match queue it pulls
ratings_next() (library items with an imdb_id and ratings_synced=0), fetches
IMDb/RT/Metacritic, applies + marks synced. So the whole library gets ratings,
not just titles you open (schema v7: ratings_synced).
- enrichment_breakdown/unmatched/retry get an 'omdb' branch (coverage =
ratings-filled, not matched). build_clients includes omdb; the lazy on-view
backfill uses the omdb worker's client.
- Dashboard orb + Manage Workers entry (★ glyph fallback where there's no logo),
yellow accent.
Seam tests: omdb worker rates the queue (ratings mode), ratings breakdown.
Next-level: real critic/audience scores beyond the TMDB star. OMDb (free key,
keyed by the imdb_id we already capture) returns IMDb / RT / Metacritic.
- OMDBClient (ratings + test); built as a non-worker 'ratings_client' on the
engine. _backfill_ratings runs in both lazy detail refreshes (overwrites, since
ratings are dynamic). schema v6: imdb_rating / rt_rating / metacritic on
movies + shows; show/movie payloads return them.
- Billboard renders branded rating badges (IMDb yellow, RT tomato/splat by
fresh/rotten, Metacritic green/yellow/red by score). Lazy refresh also triggers
when an imdb_id exists but ratings are missing.
- OMDb API-key frame in Settings (parity with TMDB/TVDB) + config GET/POST +
/enrichment/omdb/test.
Seam tests: OMDb parse, engine ratings backfill, apply_ratings + payload, config
includes omdb.
Reload bug: music's router boots first, rewrites an unknown /video-detail/... URL
to /dashboard, and my init read the already-changed URL (no restore) AND dispatched
open-detail before video-detail.js was listening (empty page + stray back button).
Fix: capture the path at SCRIPT-EVAL time (before music boots) and DEFER the
restore to a macrotask so every DOMContentLoaded handler is registered and music's
initial routing has run — then re-assert the real URL. Reload/deep-link now restore
the exact item.
Missing-episodes bug: the full-episode-list cascade only ran via the lazy refresh,
which was gated on ART being missing — so a show that already had posters/logo
never pulled its episode list (stayed owned-only). Added shows.episodes_synced
(schema v5): the worker sets it after a full cascade; show_detail returns it; the
lazy refresh now triggers when NOT synced, so owned + missing episodes populate.
Phase 3: the stylized transparent title logo replaces the text title in the
billboard (the big Plex/Netflix 'premium' jump). Sourced from TMDB images
(append_to_response=images, include_image_language=en,null) in the same detail
call — no Fanart key needed.
- schema v4: movies.logo_url / shows.logo_url (idempotent migration).
- TMDB client picks an English logo (then language-neutral, then any); enrichment
backfills logo_url gap-only; show/movie payloads return 'logo'.
- Billboard shows the logo img (with graceful fallback to the text title on error
/ when absent; title kept visually-hidden for a11y). Lazy on-view refresh now
also triggers when the logo is missing, so existing libraries fill it in.
Seam tests: English-logo pick, backfill + payload, schema.
Last capture piece, schema v3 (new people + credits tables; CREATE IF NOT EXISTS
migrates existing DBs on restart, no wipe):
- people deduped by tmdb_id; credits link to exactly one movie OR show (separate
nullable FKs + CHECK, no polymorphic id) with department/job/character/order.
- TMDB client appends credits to the detail call (free) and parses cast (name,
character, photo, billing order) + headline crew (directors/writers/creators).
- enrichment_apply backfills cast/crew gap-only (never clobbers); show/movie
detail return cast + crew. Populates on view via the existing lazy refresh-art.
- Cast & Crew section on the detail page: grouped crew line + a horizontal
cast row with circular TMDB headshots, names, characters (accent hover).
Seam tests: TMDB credit parse (+ job filtering, created_by), backfill + people
dedup across titles, gap-only no-clobber, payload shape.
Captures the richer metadata the media server already exposes (schema v2;
idempotent migrations + CREATE IF NOT EXISTS, so an existing DB upgrades on
restart with no wipe):
- movies: tagline, rating (audience), rating_critic; shows: tagline, rating,
first/last air date; episodes: still_url + rating.
- Genres as a normalised many-to-many (genres + movie_genres/show_genres link
tables — no comma-blob), deduped, replace-on-upsert.
- Plex (.genres/.tagline/.audienceRating/.rating/.thumb) + Jellyfin (Genres/
Taglines/CommunityRating/CriticRating/Premiere+EndDate/episode Primary) both
extract them; episode stills served via /api/video/poster/episode/<id>.
- Detail payloads return genres/tagline/rating/air-dates + per-episode has_still;
the billboard shows a tagline, ★ score, genre chips, and episode rows render
REAL stills (no more orange placeholder once scanned).
Seam tests for genre dedup/replace, show+episode capture, episode still ref.
Cast/crew (people + credits) is the next phase.
Foundation for the video enrichment workers, mirroring music's per-source
columns/queries on video.db.
- Schema: tmdb_match_status/tmdb_last_attempted on movies; tmdb_+tvdb_ on shows.
Idempotent ALTER-TABLE migration adds them to existing DBs on init.
- VideoDatabase helpers (service+kind -> columns map):
enrichment_next (pending first, then not_found past retry window),
enrichment_apply (sets id/status/last_attempted + whitelisted metadata,
backfill-safe), enrichment_breakdown, enrichment_unmatched (paged), and
enrichment_retry. Same shape as music's enrichment API so the shared modal can
drive it. 30 DB tests green.
The servers already matched everything to their agents — we were dropping the
IDs. Now we store them:
- Plex: parse item.guids (imdb://, tmdb://, tvdb://); Jellyfin: parse
item.ProviderIds (added ProviderIds to the requested Fields).
- Stored on movies (tmdb_id, imdb_id), shows (tvdb_id, tmdb_id, imdb_id), and
episodes (tvdb_id) via the upserts.
- Dropped the over-strict UNIQUE on movies.tmdb_id / shows.tvdb_id (same title
can legitimately live in two libraries; we dedupe on server_id). Scanner now
wraps each upsert in try/except so one bad item can't abort a scan.
Tests: guid/ProviderIds parsing + IDs persisted. 38 video-DB/scanner tests green.
Server (Plex/Jellyfin) is the source of truth, so every scanned row carries
(server_source, server_id) for upsert + stale-removal — mirroring how music
keys on server_source + ratingKey.
- schema: server_source/server_id columns on movies/shows/episodes (+ server_id
on seasons); unique (server_source,server_id) on movies/shows (multiple NULLs
allowed so wishlist rows never block).
- VideoDatabase.upsert_movie / upsert_show_tree: take normalized, server-
agnostic dicts (a Plex/Jellyfin adapter produces them — DB never touches a
media SDK), set has_file + media_files, and prune episodes/seasons the server
no longer reports.
- prune_missing(): removes top-level movies/shows the scan didn't see (cascades
clean children).
6 new tests (insert/update/file-replace, season/episode build+prune, top-level
prune); 18 video-DB tests green.
Separate SQLite file (database/video_library.db, env VIDEO_DATABASE_PATH),
fully disconnected from music — never imported by music, imports nothing from
music, no shared write lock.
Schema (database/video_schema.sql), designed to dodge the music DB's known
pain points:
- movies; shows->seasons->episodes; channels->channel_videos (YouTube as a
first-class peer); media_files (the library); downloads (queue+history);
activity feed; root_folders / quality_profiles / video_settings config.
- No polymorphic ids: media_files/downloads use separate nullable FKs + a CHECK
that exactly one owner is set; real cascades.
- Explicit external-id columns (tmdb/tvdb/imdb/youtube), no source-id blob.
- Watchlist/Wishlist/Calendar are DERIVED VIEWS over monitored + file state
(single source of truth, can't drift like music's wishlist table did).
VideoDatabase mirrors music's conventions (WAL, foreign_keys ON, 30s busy
timeout, Row factory, once-per-process init, user_version backstop) but is an
independent implementation. 13 seam tests: schema builds, CHECK constraints
reject bad rows, cascades fire, views return correct membership, KV roundtrips,
and a guard that the module imports nothing from music.
BoulderBadgeDad