Per JohnBaumb: the single state_lock serialized progress callbacks
across every source. Pre-refactor each client owned its own download
lock, so Deezer / YouTube / Tidal workers never blocked each other.
Multi-source concurrent downloads under the unified lock fought for
the same RLock on every progress update.
Replaced the engine-wide state_lock with per-source RLocks. Each
source gets its own lock, lazily created via _source_lock() on first
use (meta-lock guards the create-race). All record mutations
(add/update/update_unless_state/remove/get/iter) take only that
source's lock — Deezer progress updates no longer block Tidal writes.
Cancelled-preserve semantics still hold because cancel + worker
terminal write target the same source, so they share that source's
lock. New test pins lock independence: holding source-A's lock from
one thread does not block a write on source-B from another.
Three correctness fixes from kettui's PR review plus the web_server
migration to generic accessors.
- Engine alias map: register_plugin accepts aliases tuple; get_plugin
+ cancel_download resolve through it. Fixes deezer_dl cancels
silently routing to soulseek.
- Orchestrator hybrid_order normalization: _resolve_source_chain
routes raw config names through registry.get_spec() so legacy
deezer_dl entries don't drop deezer from hybrid mode.
- Atomic update_record_unless_state on the engine: holds state_lock
across the check + write. Both _mark_terminal AND the success path
use it now so a Cancelled state set mid-impl can't be clobbered.
- web_server.py: 30 soulseek_client.<source> reaches migrated to
client("<source>"); shutdown-check setup migrated to generic
registry iteration; 4 hifi reload sites use reload_instances('hifi').
- 18 new tests pin every fix.
Three findings from a final review pass:
1. **Worker clobbered Cancelled with Errored when impl returned
None / raised mid-cancel.** The legacy per-client thread workers
each had a guard (``if state != 'Cancelled': state = 'Errored'``);
the shared worker dropped it. Fix: new ``_mark_terminal`` helper
in BackgroundDownloadWorker reads current state before writing
the terminal one and leaves Cancelled alone. SoundCloud test
updated back to the strict Cancelled-only assertion (had been
loosened to accept Errored as a workaround). Two new pinning
tests catch the regression.
2. **Dead code in engine.py.** ``find_record`` and
``iter_all_records`` had no production callers — only tests.
Removed them. Concurrent-add stress test rewritten to use the
per-source iterator that's actually in use.
3. **Silent ``except Exception: pass`` in cross-source query
methods.** Faithful to legacy behavior (one source failing
shouldn't take down aggregation) but Cin's standard is "log
even when you swallow." Each silent-swallow site now logs at
debug level so the source name + exception are inspectable
without adding warning-level noise.
Suite still green (2049 passed).
`BackgroundDownloadWorker` lives on the engine and owns the
boilerplate every streaming download client currently
hand-rolls: thread spawn, per-source semaphore, rate-limit
delay, state lifecycle (Initializing → InProgress → Completed
or Errored), exception capture.
Plugins provide only the atomic download op (`impl_callable`).
Per-source rate-limit policy (concurrency, delay) is configured
on the worker via `set_concurrency` / `set_delay`. Source-
specific record fields merge in via `extra_record_fields` so
existing consumer code that reads `video_id`, `track_id`,
`permalink_url`, etc. keeps working post-migration. Username
slot supports override (Deezer's legacy `'deezer_dl'`).
Phase C1 scope: worker exists. No client migrated yet — C2-C7
migrate sources one at a time, each gated by the Phase A
pinning tests so per-source contract drift fails fast.
10 new tests pin the worker contract: UUID id format, initial
record shape, extra-fields merge, username override, state
transitions on success / impl-returns-None / impl-raises,
semaphore serialization (default + parallel), rate-limit
delay between successive downloads.
Suite still green (308 download tests). Pure additive.
Broque Thomas