Alongside the per-worker 'Retry all failed', the worker modal now has a topbar
'Retry all failed' that re-queues every failed/not_found item across ALL workers and
kinds in one click — one-shot recovery after an API outage left lots errored.
- db.retry_all_failed() derives the full service+kind set from the same _ENRICH /
_BACKFILL maps the workers use (tmdb/tvdb + omdb + fanart/opensubtitles/trakt/
tvmaze/anilist/wikidata + ryd/sponsorblock/dearrow), loops enrichment_retry, returns
the total re-queued. POST /api/video/enrichment/retry-all-failed.
- Topbar button (amber, text) → calls it, toasts the count, refreshes the modal.
DB test (resets across matcher + backfill + youtube service, deterministic count) +
frontend wiring test. ruff + node --check clean.
The calendar pulled every airing show in the library regardless of whether you
follow it. Now it's scoped to the EFFECTIVE watchlist by default — explicit show
follows ∪ airing library shows (not muted), same logic as the Shows watchlist tab —
so it tracks what you actually care about, and you can mute a show off it. A
'Watchlist / All library' toggle on the calendar lets you flip to everything you
own (remembered in localStorage).
- calendar_upcoming(watchlist_only=) adds the watchlist filter; /calendar takes
?scope=watchlist|all (default watchlist).
- Calendar page gets a scope toggle (defaults watchlist, persists, refetches on
change).
Tests: DB scope (followed-only / airing-default / mute drops out / all-library sees
all) + frontend wiring. node --check clean.
After a grab (manual or Auto) the user now SEES what happened and can follow it:
- The chosen result card is spotlighted (Auto scrolls it into view) and grows a live
tracker: a progress bar that follows the real download + a 'Track on Downloads ↗'
button that closes the modal and jumps to the Downloads page. Polls the new
GET /api/video/downloads/status?id= until the download reaches a terminal state.
- A movie's detail page shows a live download chip (progress bar + %) for any in-flight
download of that title; clicking it jumps to Downloads. Looks up by media identity
via /downloads/status?media_id=&media_source=, polls while active, clears on
navigate-away. (video-detail.js)
- Result CARDS redesigned (the part you didn't like): a column card with a colour
resolution badge, a green 'accepted' edge, cleaner hierarchy, and the grab button is
now an accent 'Get' pill matching the source Auto button language.
Plumbing: new lightweight /downloads/status endpoint (by id, or by media for detail
pages); soulsync:video-navigate event in video-side.js to reach a top-level page from
anywhere; VideoGet.close exported so the tracker can dismiss the modal. All video-only.
Tests: status endpoint (by id + by media + null cases) in test_video_api.py;
tracking/detail/nav wiring in test_video_download_tracking.py. node --check clean;
isolation guards green.
The video side gets its OWN automations at music-side parity, kept separate so
nothing on the music side breaks. First twin: Scan Video Library — tells the media
server to rescan the user's SELECTED video sections (movies/TV, never music), then
reads the result into video.db so freshly-downloaded media shows as owned.
Architecture (scope tags + video twins on the shared engine):
- Handler core/automation/handlers/video_scan_library.py — pure function with
injected I/O (server_refresh / run_video_scan); production lazily binds
refresh_video_server_sections() + the video scanner. Owns its own progress.
Lives on the SHARED automation side so it may import core.video (isolation only
forbids core/video & api/video from importing music, not the reverse).
- blocks.py gains a 'scope' tag ('both' generic / 'video' video-only / absent=music)
+ blocks_for_scope(). The music /api/automations/blocks now filters out video
blocks; new isolated /api/video/automations/blocks serves the video palette.
- automation_engine seeds 'Scan Video Library' (owned_by='video', schedule 6h) so
it appears ONLY on the video Automations page; ensure_system_automations now
honours owned_by + action_config. Music page excludes owned_by='video' rows.
kettui: seam-level tests for every handler path (happy/no-server/scan-error/never-
raises/mode), scope filtering (music excludes video, video gets generics, music
parity preserved), seeding (owned_by + mode), registration drift guard. 39 new
tests; full automation suite (288) + isolation guards green.
When a grabbed release fails (transfer error / peer-cancel / never lands), the engine
now retries instead of giving up — the music-style depth:
- Grab stores the OTHER accepted results as a retry pool + the search context (schema
v16: candidates / search_ctx / tried_queries / tried_files / attempts).
- core/video/retry.py (pure, tested): plan_retry() → try the next-best candidate; when
the pool is dry, next_query() generates an ALTERNATE query (movie: drop the year; TV:
numbering variants) to re-search; budget MAX_ATTEMPTS=6. merge_candidates dedupes
against already-tried releases.
- Monitor: on failure, _fail_or_retry hops to the next candidate inline; if none, flips
the row to a new 'searching' state and a background requery thread re-searches the
alternate query, evaluates against the profile, and starts the best fresh hit — or
marks failed once truly exhausted. 'searching' rows are owned by their thread.
- Page: 'Searching' status (Trying another release…) + a 'Nx' attempt badge.
16 tests (retry engine + candidate-retry transition); ruff clean on touched files.
The poll capped at 32s but slskd results trickle in over ~50s (the music side waits
the whole search_timeout), so slow searches like 'Project Hail Mary' returned 'none'
before results landed. Now:
- /search/start returns poll_ms (slskd search_timeout + 8s); the UI polls that long
(capped 80s), streaming results as they arrive, stopping early only once results
clearly plateau (≥20s + stable) or hit 25.
- /search/poll returns total_files; when 0 video releases but slskd DID return files,
the panel says 'returned N files, but none are video — likely audio/other for this
title' instead of a blank 'none' (Soulseek is audio-heavy; many movie titles are
audiobooks there). slskd_search.poll_search() returns {hits, total_files}.
Tests green, ruff + balance clean.
Grabs now carry the movie's identity so the Downloads cards are rich, not anonymous:
- video_downloads gains media_id / media_source / year / poster_url (schema v15 +
migration); grab stores them (passed from the get-modal → download view → grab).
- Cards show the POSTER in the art tile (emoji fallback), 'Title (Year)', a quality
chip (1080p · BluRay · X265), and an ↗ Open button that jumps to the movie/show
detail page (dispatches soulsync:video-open-detail). Cancel/retry unchanged.
16 tests green, ruff + balance clean.
Fixes the 'cancelled but still shows running' stuck bug and adds real depth:
- classify_state now distinguishes 'cancelled' from 'failed'.
- Monitor is robust to slskd forgetting a transfer: if it's gone, it first tries to
complete from the FILE on disk (survives the music 'Clean Completed Downloads'
auto-clear), else counts misses and fails the row after ~8 polls instead of hanging
on 'downloading' forever. Cancelled transfers → cancelled status.
- POST /downloads/cancel (slskd DELETE transfer + mark cancelled) and /downloads/retry
(re-grab the same release). get_video_download(id) added; clear includes cancelled.
process_download stays pure (fs/slskd injected); 15 tests, ruff + guard clean.
Phase B (page: tabs/queue/history/cancel+retry buttons) next.
The old search did a 4.5s slskd search + 8s wait; slskd responses trickle in over
10-30s, so the window closed before results arrived (you'd see them in slskd but the
panel said none). Now it works like the music side:
- slskd_search.start_search() (uses the shared soulseek.search_timeout) + poll_responses().
- POST /downloads/search/start (mock = immediate; soulseek = returns a search id) +
GET /downloads/search/poll. Shared _evaluate_hits ranks each poll's hits.
- UI streams: starts the search, polls every 1.3s, renders results live with a
'searching…' badge, stops when results plateau (4 stable polls) or ~32s. Live
re-renders suppress per-card entrance so it doesn't blink.
Tests green, ruff clean.
Phase 3 — the UX:
- Grab button now actually grabs (Soulseek results only — they carry the slskd
username/filename): POSTs /downloads/grab with the result + search context, shows
✓ on success, toasts 'Sent to Downloads', fires a refresh event. Endpoint returns
size_bytes for the payload.
- New Downloads PAGE (video-downloads-page.js, .vdpg-*): every grab lands here.
Polls /downloads/active while open and shows live status — type icon (🎬/📺/▶️),
title + release, a pulsing 'Downloading' pill with a shimmering progress bar, then
'Completed' with the file's library destination (→ /media/movies/…), or 'Failed'
with the reason. Clear-finished button, empty state, staggered entrance, vibes.
Reduced-motion honored. JS/CSS/HTML balance clean (verified w/ a real tokenizer).
Phase 2 — the engine:
- core/video/download_monitor.py: a daemon thread polls slskd for active video
downloads, updates progress, and on completion MOVES the file from the shared
download folder into the per-type library folder + marks it completed. The
per-download decision (process_download) is pure (fs + slskd injected) — 6 tests.
- POST /downloads/grab: validates (Soulseek-only v1), resolves the target library by
kind, starts the slskd download, records the row, lazily starts the monitor.
- GET /downloads/active (list + ensure monitor running) · POST /downloads/clear
(drop finished).
14 tests, isolation guard + ruff clean. Grab button + Downloads page next.
- Input/download folder is now the SAME shared config key the music side uses
(soulseek.download_path via config_manager) — change it on either side and both
follow, one physical download dir, simpler Docker mounts. Output libraries stay
video-specific in video.db. ZERO music code touched (read/write the shared key only).
- docker-compose: documented the shared ./downloads input mount, and added the three
video library OUTPUT mounts (./Movies→/media/movies, ./TV→/media/tv,
./YouTube→/media/youtube) matching the in-app placeholders.
- Test monkeypatches config_manager: asserts the input folder writes the shared
soulseek.download_path (music sees it) and is NOT in video.db; libraries persist to
video.db; legacy migration intact. ruff + guard + balance clean.
One shared download (input) folder feeds three separate library (output) folders —
the engine routes a finished download to the one matching its type, so YouTube never
lands in your real TV library (own Plex/Jellyfin library + agent). Replaces the single
transfer_path with movies_path / tv_path / youtube_path (legacy transfer_path migrates
into Movies on first read). Settings UI: shared Download folder + 🎬 Movies / 📺 TV /
▶️ YouTube library inputs. Tests updated incl. the migration. ruff + balance clean.
The Soulseek ⌕ now actually queries your slskd instance instead of fabricating
results. core/video/slskd_search.py (isolated): build_query(scope,…) → POST
/api/v0/searches (shared soulseek.* URL+key) → poll /responses (bounded ~8s,
early-exit at 25) → keep video files → group_video_files() folds them into one hit
per release folder with a peer count + the fastest available source. Same
{title,size_bytes,…} shape, so parse→evaluate→rank/cards are unchanged.
Endpoint routes source=='soulseek' to slskd (torrent/usenet stay mocked), surfaces
'slskd not configured' / network errors, carries username/peers/slots/filename
through for the cards + a future real Grab, and caps to 40. Pure helpers tested (4);
isolation guard + ruff clean.
mock_search now takes the source and gives Soulseek / Torrent / Usenet distinct
hits — different release-group pools, seeder magnitudes, and which qualities show
up (Soulseek drops the top remux, Usenet drops the cam, etc.), the way real
indexers differ. Endpoint forwards body.source. Fixes 'click Torrent shows the
same results as Soulseek'. +1 test.
The shared judge both the Download modal and the later-phase engine will use:
core/video/quality_eval.py (pure, isolated) — resolution_rank/resolution_label,
meets_cutoff (loose resolution target), and evaluate_owned(file, profile) →
{meets, resolution_label, reasons[]}. A copy is 'below target' when its resolution
is under the loose cutoff, or its codec is on the reject list.
Exposed as POST /api/video/downloads/evaluate (loads the stored profile, judges the
posted file). 9 tests green, isolation guard green, ruff clean.
Boulder: 'youtube should have its own quality profile — there are not many options.'
Right — YouTube is grabbed with yt-dlp, not scene/p2p releases, so the Radarr-style
ladder/cutoff/rejects/HDR-audio tiers are meaningless. Added a small, separate profile:
- core/video/youtube_quality.py (pure, isolated): max_resolution ceiling
(best…360p), video_codec (any/av1/vp9/h264, soft), container (mp4/mkv/webm),
prefer_60fps, allow_hdr. normalize/load/save → video.db youtube_quality_profile.
- api/video/downloads.py: GET/POST /downloads/youtube-quality.
- UI: a 'YouTube Quality' card on the Downloads tab (data-video-only) — resolution
dropdown + codec/container segmented + 60fps/HDR checks, reusing the vq-* styles.
Wired into onPageShown + the save-all chain.
The (later-phase) yt-dlp downloader maps these to a format/format_sort selection.
9 new tests green, isolation guard green, ruff clean, JS/HTML balance clean.
The slskd CONNECTION settings (URL, API key, search timeout + buffer, min delay, min
peer speed, max peer queue, download timeout, auto-clear) are genuinely shared — one
slskd instance serves both sides — so the video Downloads tab surfaces them and they
read/write the app-wide config_manager soulseek.* keys, NOT video.db. Changing them
on the video side changes them for Music too (labeled 'shared with Music').
Deliberately EXCLUDES the music download/transfer paths and source mode/quality — those
stay video-specific (video.db, phases 1-3). The block shows only when soulseek is the
active/primary source (or in the hybrid chain). The 'Indexers & Downloaders' tab is
already fully shared (no video override — only data-stg='downloads' is hidden).
- /api/video/downloads/slskd GET/POST over config_manager (config.settings — shared app
config, not music code; the isolation test still passes).
- video-settings.js loadSlskd/saveSlskd (minutes↔seconds for the timeout), gated by
soulseekActive(), wired into onPageShown + the save chain.
68 video API tests green (incl. the no-music-imports guard + a shared-slskd test that
asserts it writes soulseek.* but never the video paths). Completes the Downloads-tab
settings batch (folders, quality profile, source/hybrid, shared slskd).
Video-specific Download Source section: a mode dropdown limited to Soulseek / Torrent /
Usenet / Hybrid (no streaming sources — those are music-only). In hybrid mode, a chain
builder lists the three sources with arrow-reorder + enable toggles (best-first), and
NO album-level/track-level badges (a music-only concept, per spec).
- core/video/download_config.py: pure normalize for download_mode + hybrid_order
(validates to the 3 sources, dedupes, never empties); stored in video_settings.
- /api/video/downloads/config GET/POST now carries download_mode + hybrid_order
alongside the folders.
- video-settings.js: dropdown + hybrid rows render/reorder/toggle, show the hybrid
container only in hybrid mode, save on change. Reuses the .vq-row styling.
7 tests (6 pure + 1 API). Next: the shared slskd connection block (reads/writes the
music config_manager so both sides share one slskd) + confirming the shared Indexers tab.
Foundation for the isolated video download settings. The Downloads tab is almost
entirely music-specific, so on the video side the music download sections are hidden
(video-side.css) and a data-video-only 'Video Download Folders' section takes their
place — an input (download) and output (transfer/library) folder, stored SEPARATELY
from the music soulseek.* paths in video.db's video_settings KV table.
- api/video/downloads.py: GET/POST /api/video/downloads/config (download_path,
transfer_path), registered in the video blueprint. Imports nothing from music.
- video-settings.js: loadDownloads/saveDownloads wired into onPageShown + the
video save-button chain.
- The shared 'Indexers & Downloaders' tab is left untouched (identical for both).
2 tests (round-trip + the isolation guard). Quality profile, video hybrid, and the
shared slskd block are the next phases.
Fifth/final service. Keyless, movies + shows, on by default → toggle in the
'Community Data (No Key)' frame.
- WikidataWorker: two-step lookup — find the entity by IMDb id (haswbstatement P345)
then read its official website (P856); stores wikidata_url. Registered in
build_backfill_workers.
- DB: wikidata_url/status/attempted on movies + shows + _BACKFILL/_BACKFILL_COLS;
both detail payloads return wikidata_url.
- config GET/POST wikidata_enabled; manager orb (green 🔗) + status poll; detail page
'Official Site' link badge alongside IMDb/TMDB/TVDB.
- 3 new tests (incl. the two-step fetch) + fixed-set/config assertions.
34 backfill tests green, ruff clean. All five new services (Trakt, TVmaze, AniList,
DeArrow, Wikidata) now have the full worker/DB/connections/orb/manager/detail parity.
Fourth service. Rides the YouTube-video enrich path (like RYD/SponsorBlock), keyless,
on by default → toggle in the YouTube-Extras frame.
- DeArrowWorker: fetches the branding API and stores the first non-original crowd
title for a cached YouTube video. apply_youtube_dearrow + youtube_video_dearrow_title
DB methods; dearrow_status added to the youtube_enrich next/breakdown whitelists.
- Schema: dearrow_title/status/attempted on youtube_video_stats (video_schema.sql +
_COLUMN_MIGRATIONS for existing DBs).
- config GET/POST dearrow_enabled; manager orb (blue 🏹, video-kind) + status poll;
YT video-detail panel shows a 'DeArrow' alt-title (payload via youtube.py + CSS).
- 4 new tests + fixed-set/config assertions.
30 backfill tests green. (My change is ruff-clean; the 12 pre-existing S110s in
api/video/youtube.py are unrelated tech debt on this branch, left untouched.)
Third service. Keyless GraphQL (new _http_post_json helper) → enable toggle in the
'Community Data (No Key)' frame, OFF by default (anime-niche + title-search match).
- AniListWorker: TV-only, searches AniList by title and stores the anime averageScore
(0-100), with a conservative normalized-title guard so a fuzzy anime search can't
attach a score to a non-anime show. anilist_enabled toggle (default off).
- DB: anilist_score/status/attempted on shows + _BACKFILL/_BACKFILL_COLS (keyed on
title); show_detail returns anilist_score.
- config GET/POST anilist_enabled (default 0); manager orb (blue 🎌) + status poll;
detail page 'AniList 85%' chip (+ CSS).
- 4 new tests (incl. the title-mismatch rejection) + fixed-set/config assertions.
27 backfill tests green, ruff clean. (Note: the youtube_status_route test still
flaps on the sandbox WSL WAL disk-I/O — environmental, unrelated.)
Second service. Keyless (no API key) → an enable toggle in a new 'Community Data
(No Key)' connections frame, mirroring the YouTube-Extras pattern.
- TVmazeWorker: TV-only (no movie DB), looks a show up by imdb/thetvdb id and
gap-fills the TVmaze community rating. On by default; tvmaze_enabled toggle.
- DB: tvmaze_rating/status/attempted on shows + _BACKFILL/_BACKFILL_COLS (show only);
show_detail returns tvmaze_rating.
- config GET/POST tvmaze_enabled; manager orb (teal 📺, show-kind) + status poll;
detail page TVmaze rating chip (+ CSS).
- 4 new tests + the 3 fixed-set/config-exact assertions updated.
Note: one UNRELATED test (youtube_status_route) flaps on a WSL WAL 'disk I/O error'
in this sandbox (pass/fail/fail across reruns of identical code) — environmental, not
this change; the TVmaze + config tests pass consistently.
First of the new enrichment services, wired to the SAME standard as the rest:
- TraktWorker backfill (core/video/enrichment/backfill.py): looks a title up by its
IMDb id (?extended=full) and gap-fills the community rating + vote count. Registered
in build_backfill_workers.
- DB: trakt_rating/trakt_votes/trakt_status/trakt_attempted columns + _BACKFILL /
_BACKFILL_COLS registration; detail payloads return trakt_rating/votes.
- Connections tab: Trakt service frame (Client ID field + Test button), wired into
video-settings.js load/save/bindings and the /enrichment/config GET+POST.
- Worker-manager modal + orb: WORKERS registry entry (red ★) + status-poll SERVICES
list, so it gets the same orb animation + per-service matched/pending/error card.
- Detail page: a 'Trakt 8.2' rating chip alongside IMDb/RT/Metacritic (+ CSS).
5 new tests + 2 fixed-set assertions updated; 249 video tests green, ruff clean.
- YouTube video cards show 👍/👎 like/dislike counts (from Return YouTube Dislike)
next to views, threaded through ytEpisodeOf.
- New GET /youtube/video/<id>/segments exposes stored SponsorBlock segments so the
player can offer skips. Update config-shape test for the new keys/toggles.
- Manage Workers modal: register fanart.tv / OpenSubtitles / YouTube Votes (RYD)
/ SponsorBlock in the worker rail (cards + animations + pause/resume come free
via the shared .em-* design); add the 'video' entity kind.
- Settings: fanart.tv + OpenSubtitles API-key fields (with Test buttons) and a
no-key 'YouTube Extras' toggle frame (RYD + SponsorBlock on/off).
- API /enrichment/config now reads/writes the two keys + the two toggles; a key
change rebuilds the engine so the worker turns on immediately.
Correcting my earlier 'YouTube caps at ~200' claim — that's the ANONYMOUS ceiling.
YouTube throttles unauthenticated playlist extraction to ~100 (yt-dlp) / ~200
(InnerTube); ytdl-sub gets everything because it runs with browser cookies. Our
code already wires cookiesfrombrowser via youtube.cookies_browser (same as the
music client) — it's just unset. The detail endpoint now resolves with a high
limit so a cookie-authenticated yt-dlp pages the WHOLE playlist, and uses whichever
source (yt-dlp vs InnerTube) returned more, with the true header count. The partial
note now tells you to add cookies in Settings to load the full list.
A big playlist read '97' because yt-dlp flat caps playlist extraction at ~100 and
its playlist_count is often unset, so we showed the FETCHED count. Now the detail
endpoint overlays the InnerTube playlist browse (browseId VL<id>): curator-ordered
videos up to ~200 (vs yt-dlp's ~100) AND the real count from the header's
numVideosText (e.g. '512 videos'). The billboard shows the true total; when the
list is partial it says 'Showing the first N of TOTAL videos (YouTube limits
playlist browsing).' Live-validated (Veritasium uploads: total 512, 200 loaded).
103 tests green.
Honest cap: YouTube's browse only exposes ~200 of a playlist (page 2 returns no
continuation), so very large playlists show the right COUNT but list ~200.
The channel card's 'N videos' was actually the count of WISHED videos (0 until you
wish some) — so every channel read '0 videos'. It now shows the REMEMBERED catalog
size (from youtube_channel_videos, which fills in as a channel is enriched/opened),
falling back to 'Channel' when nothing's cached yet. Wished count is kept as a
separate wished_count field. Playlists likewise show their video count (cached when
the playlist is followed-with-videos or opened) instead of a static 'Playlist'.
list_watchlist_channels/playlists return the remembered count; the playlist detail +
follow endpoints cache the list so the count is known. 138 tests green.
Channel playlists (the collapsible sections on a channel page) now each carry the
standard watchlist button — Add to Watchlist / In Watchlist — so you can follow a
channel's playlist without pasting its link. The /youtube/playlists/<id> endpoint
flags each playlist with its follow state to hydrate the button; the button click
is handled before the row's expand toggle (stopPropagation), and adds via
followPlaylist — the same plumbing the search chip + watchlist use. Followed ones
then appear in the watchlist Channels tab alongside channels.
A playlist is now a first-class watchlist type alongside channels:
- core: parse_playlist_id (URL/bare id; rejects mixes RD… + personal lists) +
resolve_playlist → {playlist_id, title, channel_title, video_count, thumbnail,
videos} in the CURATOR's order (partial set, not by year).
- db: add/remove/list/watch_state for kind='playlist' video_watchlist rows
(mirrors channels; same surrogate scheme, separate kind so they don't collide).
- api: /youtube/resolve now detects playlist links; /youtube/playlist/follow +
/unfollow; the existing /youtube/playlist/<id> is upgraded to also return the
playlist meta + following (serves the channel-page expansion AND the new detail
view from one route); /youtube/channels also returns followed playlists.
Validated live (Lex Fridman playlist resolves title+owner+videos). 190 tests green.
Frontend (search detect + result card + detail view + watchlist display) next.
Each /videos batch was 3 InnerTube pages + 0.4s sleeps (>1s → tripped the
slow-request WARNING) and carried the ~2KB continuation token in the URL, so
every batch dumped a giant warning line — a dozen per channel open. Now it's a
POST (token in the body, never the URL) fetching ONE page per call with no
server sleep, so each request is fast (<1s, no warning) and the frontend's 120ms
pacing keeps it polite. 17 youtube API tests green.
Channel pages re-fetched everything every open (re-stream + ~3s yt-dlp metadata).
Now we remember what we learn about a channel and serve it cache-first:
- schema: youtube_channel_videos (list) + youtube_channel_meta (avatar/subs/tags/
banner); dates stay in youtube_video_dates, merged on read. DB: cache_/get_
channel_videos + cache_/get_channel_meta (upserts COALESCE so refreshes never
drop fields).
- enricher: _enrich now REMEMBERS a channel — caches the full InnerTube catalog
(list, via new innertube_channel_catalog) + metadata + dates, not just dates.
Since it already sweeps followed channels, watchlisted channels get pre-warmed
in the background → opening them is instant.
- channel endpoint: cache-first. A remembered channel renders from cache with NO
network (returns from_cache:true); a miss resolves live (yt-dlp) and remembers
it. The /videos batch endpoint caches every page it streams.
- frontend: cache hit renders the full catalog instantly, then re-streams to
refresh QUIETLY (new uploads/date fixes); only the first (miss) load shows the
'loading full history' banner.
Live-validated (MrBeast catalog: 90/3 pages, all titled+dated). 190 tests green.
The channel page was capped at the recent 90 (yt-dlp flat, hard min(90,limit)),
so prolific channels (Ninja Kidz etc.) showed only ~90 of hundreds. Now it loads
everything via YouTube's InnerTube API, paginated by continuation token — each
page fetched once (O(n), light on rate limits), unlike yt-dlp offset batches
which re-scan from the start every time.
- core: innertube_parse_video_items (keeps title+thumbnail, not just dates) +
innertube_channel_videos_page(channel_id, continuation) → {videos, continuation}.
- api: GET /youtube/channel/<id>/videos?continuation=<tok> returns ~a batch of
videos (approx dates, refined from the date cache) + the next token.
- frontend: ytLoadAllVideos streams batches after the fast initial 90 render,
folding each into the year-seasons live — backfills dates on the videos already
shown AND appends older ones, re-rendering per batch (episode grid only when the
viewed season changed). Replaces the old date-only re-poll (this dates AND
expands). Safety ceiling 2000; a 'Loading full history… N so far' indicator.
Validated live (MrBeast: 30/page, clean continuation, all dated). 101 tests green.
Bug from the logs: opening MrBeast showed only '2026: 3, Earlier: 87' because
the InnerTube date enrichment only fired for FOLLOWED channels — MrBeast was
opened, not followed, so it never ran and the page had only RSS dates (and RSS
includes Shorts that don't match the Videos tab → ~3 real matches). Now opening
any channel page enqueues enrichment, so its full catalog gets dated in the
background and the years populate via the existing re-poll. (24h gate still
prevents re-sweeping; the enqueue still no-ops under tests.)
Two gaps from real use:
- When all proxy instances are down (common), the fallback only dated WISHED
videos, leaving most as 'Earlier videos'. Now on proxy failure it flat-resolves
the channel's recent uploads and dates THOSE via yt-dlp (throttled, cached) —
so years populate fully even with no working proxy.
- A channel followed before the enricher existed was never queued ('nothing to
process'). /youtube/channels now sweeps: any followed channel not enriched
recently gets enqueued — so viewing the Channels tab (or the wishlist badge
refresh) picks up existing follows.
The detail page still reads cache+RSS (fast); the enricher fills the cache. 51
tests green.
- The standalone enricher now reports orb telemetry: stats()/pause()/resume();
/enrichment/youtube/status (+pause/resume) special-cased in the route. Added
the 4th header worker button (red ▶), WORKER_DEFS entry, and SERVICES wiring —
it's not on the socket so video-enrichment.js polls it every 3s. It animates
(idle → active orb + inbound pulses) while a followed channel's dates are being
fetched, like the TMDB/TVDB/OMDb orbs.
- Channel detail now uses the SAME standard watchlist button as shows/movies
(library-artist-watchlist-btn, ✓/＋ icon, 'In Watchlist'/'Watchlist') instead
of a bespoke 'Follow' — consistency. Still wired to the channel follow action.
176 backend tests green; JS balanced; music untouched.
Follow a channel → a background job fetches its FULL upload-date catalog so the
channel page's year-seasons populate fully, cached permanently (one-time per
channel, instant after).
- core/video/youtube.py: proxy_channel_dates() — no-key BULK dates via Piped/
Invidious instances (tries several, paginates); parse_proxy_dates handles both
shapes. The clean path the user wanted.
- core/video/youtube_enrichment.py: YoutubeDateEnricher daemon — enqueue(channel)
→ proxy bulk → cache; per-video yt-dlp only as a throttled fallback (cap 60,
0.4s) for wished videos when every proxy is down. Skips channels enriched in
the last 24h.
- db: youtube_channel_enrichment table + mark/check + wishlisted_video_ids_for_
channel. Enqueued from /youtube/follow and on opening a followed channel.
Scoped to FOLLOWED channels only (per request). 174 tests green; enricher imports
nothing from music.
Flat listing has no upload dates, so channels showed one 'All Videos' season.
Now real year-seasons, filled from cheap sources and cached so they grow:
- core: parse_rss_dates / channel_recent_dates — one public RSS GET dates the
~15 most-recent uploads (no yt-dlp, no bot risk).
- db: youtube_video_dates cache table + cache_video_dates / get_video_dates.
- /youtube/channel merges cached + RSS dates onto the flat videos (year-seasons)
and caches what it learns; /youtube/video caches each fetched date too — so
expanding/wishing videos progressively fills the catalog's years, instant on
repeat. Dateless tail groups as 'Earlier videos'. Missing-only toggle hidden
for channels. 84 db + 84 youtube/api tests green.
Full day-one coverage still needs the YouTube Data API's publishedAt (optional,
yt-dlp can't match it cheaply) — RSS + cache is the no-key best.
Screenshot showed channel results with empty circles, not the initials chip —
i.e. an avatar URL was present but the image was failing to load. Two causes
addressed:
- The image proxy fetched googleusercontent/yt3 with NO User-Agent, which the
CDN 403s for some avatars → blank. Now sends a browser UA + Accept header.
- A failed avatar now swaps to the initials chip (onerror → outerHTML) instead
of hiding the img and leaving an empty circle. Also proxy protocol-relative +
deeper-subdomain YouTube image hosts. img-proxy tests green.
A text search now also surfaces matching YouTube channels (not just the paste-a-
link path). search_channels() extracts the channels-filtered YouTube results
page (flat); API GET /youtube/search hydrates a 'following' flag. The search
page fires it in parallel with the TMDB search and appends a 'YouTube channels'
group when it returns (best-effort; never blocks/breaks TMDB results). Cards open
the in-app channel page. 82 youtube+api tests green; balanced; music untouched.
The channel avatar (and video thumbnails) are yt3.googleusercontent.com /
i.ytimg.com URLs; hotlink/CORS policy could blank them (failed <img> hides on
the watchlist, falls to initials on the wishlist orb) — which is why the poster
'vanished' on both pages.
Extend the /api/video/img proxy allowlist to YouTube CDN hosts (ytimg.com /
ggpht.com / googleusercontent.com, https-only, still SSRF-safe) and route all
YouTube art through it: VideoYoutube.img() helper used by the watchlist channel
cards, search chip, the wishlist nebula orb/season/still art, and the channel
detail avatar/banner/thumbnails. 2 img-proxy tests green; JS balanced.
Flat channel listing doesn't always surface the avatar, so it could be stored
null → the orb fell back to plain initials (looked like a missing poster). Two
fixes:
- Orb falls back to the channel's newest video thumbnail when the avatar is
absent, so it's never blank.
- Opening the channel page (which resolves the real avatar) now backfills it
onto every wished row via set_wishlist_channel_poster — so the actual channel
avatar appears on the wishlist orb thereafter. 6 tests green.
Flat channel listing can't return descriptions, so selecting a video showed 'No
description'. Now we do a non-flat single-video extract on demand — the way the
TV nebula lazy-loads guest stars:
- core: video_detail(id) / shape_video() — description, views, likes, duration,
tags, channel, webpage_url (full extract, yt-dlp injectable for tests).
- API: GET /youtube/video/<id> — returns it AND backfills the description onto
the wishlist row (set_wishlist_video_overview), so re-opening is instant.
- Wishlist info bar (youtube): on select, lazy-fetch → real description +
eyebrow (date · duration · views) + a 'Watch on YouTube' link; cached on the
episode object. 'Loading details…' while in flight.
Mirrors the episode lazy-load + art-backfill patterns. 73 youtube+api tests
green; brace/CSS balanced; music untouched.
- Resolver: capture banner_url + separate avatar (by thumbnail id), keeping
per-video duration/views for the detail hero.
- query_youtube_wishlist reshaped to the EXACT TV-nebula shape: channel = show,
upload YEAR = season (newest first), video = episode (newest=ep1 within a
year), carrying source='youtube' + youtube_id + per-video source_id. Surrogate
int returned as tmdb_id so the nebula's int keying just works.
- New API: GET /youtube/channel/<id> (full channel detail — meta + deeper
uploads + following + per-video wished flags) and POST /youtube/wishlist/add
(per-video wish). DB: youtube_video_wish_state, remove_one_video_from_wishlist.
- Tests updated for the nebula shape + banner + detail endpoints. 82 DB + 69
api/youtube tests green.
api/video/youtube.py wired into the blueprint:
- GET /youtube/resolve — preview a pasted channel (meta + recent uploads) +
a 'following' hydration flag; no writes.
- POST /youtube/follow — {url} or pre-resolved {channel} → follow + wish its
recent uploads (capped 30). Returns channel + counts.
- POST /youtube/unfollow — {youtube_id} → un-follow (keeps wished videos).
- GET /youtube/channels — followed channels for the watchlist page.
- GET /youtube/wishlist — wished videos grouped by channel.
- POST /youtube/wishlist/remove — scope channel|video.
yt-dlp call lives behind resolve (injectable/mockable). 38 API tests green.
BoulderBadgeDad