readme update

README.md

@@ -16,12 +16,13 @@
 
 SoulSync bridges streaming services to your media server with automated discovery:
 
-1. **Monitors artists** → Automatically detects new releases
+1. **Monitors artists** → Automatically detects new releases from your watchlist
 2. **Generates playlists** → Release Radar, Discovery Weekly, Seasonal, Decade/Genre mixes
-3. **Downloads missing tracks** → From Soulseek, Beatport charts, playlists
-4. **Enriches metadata** → LRC lyrics, album art, proper tags
-5. **Organizes files** → Custom templates for clean folder structures
-6. **Syncs media server** → Plex, Jellyfin, or Navidrome stay updated
+3. **Downloads missing tracks** → From Soulseek, YouTube, Beatport charts, Spotify/Tidal/YouTube playlists
+4. **Verifies downloads** → AcoustID fingerprinting confirms you got the right track
+5. **Enriches metadata** → LRC lyrics, album art, MusicBrainz IDs, proper tags
+6. **Organizes files** → Custom templates for clean folder structures
+7. **Syncs media server** → Plex, Jellyfin, or Navidrome stay updated automatically
 
 ---
 
@@ -40,61 +41,94 @@ SoulSync bridges streaming services to your media server with automated discover
 - Built from 1000+ track discovery pool
 - Refreshes every 24 hours
 
-**Seasonal Playlists** - Halloween, Christmas, Valentine's, Summer, Spring, Autumn (auto-generated)
+**Seasonal Playlists** - Halloween, Christmas, Valentine's, Summer, Spring, Autumn (auto-generated based on time of year)
 
 **Personalized Playlists** (12+ types)
 - Recently Added, Top Tracks, Forgotten Favorites
 - Decade Playlists (1960s-2020s), Genre Playlists (15 categories)
-- Daily Mixes, Hidden Gems, Popular Picks, Custom Builder
+- Daily Mixes, Hidden Gems, Popular Picks, Discovery Shuffle, Familiar Favorites
+- Custom Playlist Builder
 
 **ListenBrainz** - Import recommendation and community playlists
 
-**Beatport** - Electronic music charts by genre (House, Techno, Trance, etc.)
-- Top 100, Hype Charts, DJ Charts, Staff Picks
+**Beatport** - Full electronic music integration with genre browser (39+ genres)
+- Top 100, Hype Charts, DJ Charts, Staff Picks, New Releases
+- Browse by genre with dedicated discovery UI
 
 ### Multi-Source Downloads
 
-**Sources**: Soulseek (FLAC priority), YouTube (Audio), Beatport charts, Spotify/Tidal playlists
-
-**Features**
-- Quality profiles: Audiophile, Balanced, Mobile
-- Automatic format fallback (FLAC → MP3)
-- Duplicate prevention against library
-- Batch processing with retry logic
-- Synchronized lyrics (LRC) for every track
-
-### Metadata & Reliability
-
-**Dual-Source System**
-- **Primary**: Spotify (Preferred for richer data and discovery features)
-- **Backup**: iTunes (No authentication required)
-- **Redundancy**: System automatically manages both sources. If Spotify is authorized, it is prioritized. If Spotify is unavailable, rate-limited, or unauthorized, SoulSync **seamlessly switches to iTunes** for metadata, cover art, and artist tracking.
-- **Fail-Safe**: Even with Spotify authorized, iTunes metadata is maintained as a redundant layer to ensure zero downtime.
+**Download Sources**: Soulseek (FLAC priority), YouTube (audio extraction), or Hybrid mode (tries primary source, falls back automatically)
 
+**Playlist Sources**: Spotify, Tidal, YouTube, Beatport charts, ListenBrainz
 
-### Advanced Matching
-
-- Unicode/accent handling (KoЯn, Björk, A$AP Rocky)
-- Fuzzy matching with confidence scoring
-- Album variation detection (Deluxe, Remastered, etc.)
-- Multi-strategy: exact → normalized → fallback
+**Features**
+- Quality profiles with presets: Audiophile, Balanced, Space Saver
+- Per-format configuration (FLAC, MP3, OGG, AAC, WMA) with min/max size and priority
+- Automatic format fallback (FLAC → MP3 → next best)
+- Duplicate prevention against existing library
+- Batch processing with concurrent workers and automatic retry logic
+- Source reuse — when downloading an album, reuses the same uploader for consistency
+- Synchronized lyrics (LRC) via LRClib for every track
+
+### Audio Verification
+
+**AcoustID Fingerprinting** (optional) - Verifies downloaded files actually match the expected track
+- Generates audio fingerprints and checks against AcoustID database
+- Compares title and artist using fuzzy matching with configurable thresholds
+- Fail-open design: verification errors never block downloads, only confident mismatches are rejected
+- Mismatched files are flagged and can be re-queued via the wishlist
+
+### Dual-Source Metadata
+
+- **Primary**: Spotify — richer data, discovery features, playlist sync
+- **Backup**: iTunes/Apple Music — no authentication required, works out of the box
+- **Seamless switching**: If Spotify is unavailable, rate-limited, or unauthorized, SoulSync automatically falls back to iTunes for metadata, cover art, and artist info
+- **MusicBrainz enrichment** — background worker matches artists, albums, and tracks to MusicBrainz IDs with 90-day caching
+
+### Advanced Matching Engine
+
+- Version-aware matching: strictly rejects remixes when you want the original (and vice versa)
+- Unicode and accent handling (KoЯn, Bjork, A$AP Rocky)
+- Fuzzy matching with weighted confidence scoring (title, artist, duration)
+- Album variation detection (Deluxe, Remastered, Taylor's Version, etc.)
+- Multi-query search strategy: generates 4-6 query variations per track
+- Short title protection: prevents "Love" from matching "Loveless"
+- Source-specific weighting: Soulseek prioritizes artist match, YouTube prioritizes title match
 
 ### Automation
 
-**Watchlist** - Monitor unlimited artists, auto-discover similar artists via music-map.com
+**Watchlist** - Monitor unlimited artists with per-artist configuration
+- Choose which release types to track: Albums, EPs, Singles
+- Content filters: Live, Remixes, Acoustic, Compilations
+- Auto-discover similar artists via music-map.com
+- Periodic scanning for new releases
 
-**Wishlist** - Failed downloads retry every 30 minutes automatically
+**Wishlist** - Failed downloads are automatically queued for retry
+- Auto-processing on a configurable timer
+- Organized by category (albums vs singles)
+- Cleanup tool removes tracks you've since acquired
 
-**Background Tasks** - Database sync, discovery pool updates, seasonal content
+**Background Tasks** - Database sync, discovery pool updates, seasonal content, watchlist scanning
 
 ### Library Management
 
-- **Quality Scanner** - Find low-bitrate files to replace
-- **Duplicate Cleaner** - Identify redundant tracks
-- **Completion Tracking** - Album progress percentages
-- **Enhanced Search** - Unified search across Spotify, library, Soulseek
+- **Dashboard** with service status, system stats, and activity feed
+- **Database Updater** - Incremental or full sync from your media server (smart early-stopping for speed)
+- **Metadata Updater** - Refresh artist photos, genres, and album art from Spotify/iTunes
+- **Quality Scanner** - Find low-bitrate files that could be replaced with higher quality
+- **Duplicate Cleaner** - Identify and remove redundant tracks, reclaim disk space
+- **Completion Tracking** - See album/EP/single progress percentages per artist
+- **Enhanced Search** - Unified search across Spotify, your library, and Soulseek simultaneously
+- **Import System** - Drop existing audio files into a staging folder, match them to albums, and process with full metadata
 - **Template Organization** - `$albumartist/$album/$track - $title` (fully customizable)
 
+### Built-in Streaming Player
+
+- Stream any track directly from Soulseek before downloading
+- HTML5 audio player in the sidebar with play/pause, seeking, volume control
+- Format detection with browser compatibility checking
+- Retry logic with exponential backoff for reliability
+
 ---
 
 ## Installation
@@ -107,7 +141,16 @@ docker-compose up -d
 # Access at http://localhost:8008
 ```
 
-### Python
+### Unraid
+
+SoulSync is available as an Unraid template. Install from Community Applications or manually add the template from:
+```
+https://raw.githubusercontent.com/Nezreka/SoulSync/main/templates/soulsync.xml
+```
+
+PUID/PGID are exposed in the template — set them to match your Unraid permissions (default: 99/100 for nobody/users).
+
+### Python (No Docker)
 
 ```bash
 git clone https://github.com/Nezreka/SoulSync
@@ -124,7 +167,7 @@ python web_server.py
 ### Prerequisites
 
 - **slskd** running and accessible ([Download](https://github.com/slskd/slskd/releases))
-- **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard))
+- **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard)) — optional but recommended
 - **Media Server** (optional): Plex, Jellyfin, or Navidrome
 
 ### Step 1: Set Up slskd
@@ -188,10 +231,12 @@ Open SoulSync at `http://localhost:8008` (or your server's IP) and go to Setting
 
 **Paths:**
 - **Download Path**: This should be the container path where slskd's downloads are mapped. If you followed the volume mapping above, that's `/app/downloads`
-- **Transfer Path**: This is where SoulSync puts your organized/renamed music. The default is `/app/Transfer` - just make sure you have a volume mapping for it
+- **Transfer Path**: This is where SoulSync puts your organized/renamed music. The default is `/app/Transfer` - just make sure you have a volume mapping for it so the files persist and your media server can see them
+- **Staging Path**: Optional — for importing existing audio files. Map a host folder to `/app/Staging` if you want to use the import feature
 
 **Media Server** (optional):
 - **Plex/Jellyfin/Navidrome URL**: Use your machine's actual IP address, not `localhost`. For example `http://192.168.1.100:32400` for Plex. Same reason as slskd - `localhost` inside Docker means the container, not your host machine.
+- After connecting, run a **Database Update** from the Dashboard to populate your library
 
 ### Step 4: Docker Path Mapping
 
@@ -206,6 +251,7 @@ This is where most issues come from. The key concept: SoulSync runs inside a Doc
 | Database | `/app/data` | Use a named volume (see below) |
 | slskd Downloads | `/app/downloads` | Same folder slskd downloads to |
 | Music Library/Transfer | `/app/Transfer` | Where you want organized music to go |
+| Import Staging | `/app/Staging` | Optional — folder for importing existing files |
 
 **Example docker-compose volumes for Linux:**
 ```yaml
@@ -215,6 +261,7 @@ volumes:
   - soulsync_database:/app/data
   - /path/to/slskd/downloads:/app/downloads
   - /path/to/music/library:/app/Transfer
+  - ./staging:/app/Staging
 ```
 
 **Example for Unraid:**
@@ -225,6 +272,7 @@ volumes:
   - soulsync_database:/app/data
   - /mnt/user/Music/Downloads:/app/downloads
   - /mnt/user/Music/Library:/app/Transfer
+  - /mnt/user/appdata/soulsync/staging:/app/Staging
 ```
 
 **Important:** The database should use a named volume (`soulsync_database:/app/data`), not a direct host path mount. Mounting a host folder to `/app/database` will overwrite Python module files the app needs to run.
@@ -233,7 +281,11 @@ The paths you enter in SoulSync's settings page should always be the **container
 
 ### Unraid Users
 
-If you're using the Unraid template from Community Applications, double check that the container path for your music share is set to match what SoulSync actually uses. The download folder should map to `/app/downloads` and your library/transfer folder should map to `/app/Transfer`. Then use those same container paths in SoulSync's settings page.
+If you're using the Unraid template from Community Applications:
+- PUID/PGID are shown during setup — set them to match your permissions (default 99/100)
+- The Music Share mount gives SoulSync access to your downloads and library
+- Make sure the Transfer Dir path is **writable** — read-only mounts will cause post-processing failures
+- After installing, run a **Database Update** from the Dashboard to populate your library
 
 ---
 
@@ -241,13 +293,13 @@ If you're using the Unraid template from Community Applications, double check th
 
 **Perfect for:**
 - Self-hosters with Plex/Jellyfin/Navidrome
-- Music enthusiasts with 500+ album collections
+- Music enthusiasts building and maintaining large collections
 - Electronic music fans (Beatport integration)
-- Former Spotify users wanting local discovery
+- Former Spotify users wanting local discovery that actually works
 
 **Not ideal for:**
-- Casual users wanting simple sync
-- Slow/metered internet connections
+- Casual users wanting simple one-click sync
+- Slow or metered internet connections
 - Users uncomfortable with APIs or Docker
 
 ---
@@ -256,36 +308,41 @@ If you're using the Unraid template from Community Applications, double check th
 
 | Feature | SoulSync | Lidarr | Headphones | Beets |
 |---------|----------|--------|------------|-------|
-| Custom Discovery Algorithm | ✓ | ✗ | ✗ | ✗ |
-| Personalized Playlists (12+) | ✓ | ✗ | ✗ | ✗ |
+| Custom Discovery Playlists (12+) | ✓ | ✗ | ✗ | ✗ |
+| Seasonal/Personalized Playlists | ✓ | ✗ | ✗ | ✗ |
 | Beatport Integration | ✓ | ✗ | ✗ | ✗ |
 | ListenBrainz Playlists | ✓ | ✗ | ✗ | ✗ |
-| Multi-Source (Spotify/Tidal/YouTube) | ✓ | ✓ | ✗ | ✗ |
-| Watchlist Monitoring | ✓ (100+) | ✓ | ✓ | ✗ |
+| Multi-Source Playlists (Spotify/Tidal/YouTube) | ✓ | ✗ | ✗ | ✗ |
+| Soulseek Downloads | ✓ | ✗ | ✗ | ✗ |
+| YouTube Downloads | ✓ | ✗ | ✗ | ✗ |
+| Audio Fingerprint Verification | ✓ | ✗ | ✗ | ✓ |
+| Watchlist Monitoring | ✓ | ✓ | ✓ | ✗ |
 | LRC Lyrics | ✓ | ✗ | ✗ | Plugin |
-| Advanced Matching | ✓ | ✗ | ✗ | ✓ |
+| Version-Aware Matching | ✓ | ✗ | ✗ | ✗ |
 | Quality Scanner + Duplicate Cleaner | ✓ | ✗ | ✗ | ✓ |
 | Template-Based Organization | ✓ | ✗ | ✗ | ✓ |
-| Seasonal Playlists | ✓ | ✗ | ✗ | ✗ |
-
-**SoulSync is the only tool combining intelligent discovery with multi-source automation and library management.**
+| Built-in Streaming Player | ✓ | ✗ | ✗ | ✗ |
+| Import Existing Files | ✓ | ✓ | ✗ | ✓ |
+| Web UI | ✓ | ✓ | ✓ | ✗ |
 
 ---
 
 ## Architecture
 
-**Scale**: 83,000+ lines Python, 120+ API endpoints, handles 10,000+ album libraries
+**Scale**: ~100,000 lines across Python backend and JavaScript frontend, 60+ API endpoints, handles 10,000+ album libraries
 
-**Integrations**: Spotify, Tidal, YouTube, Plex, Jellyfin, Navidrome, Slskd, ListenBrainz, LRClib, music-map.com, Beatport
+**Integrations**: Spotify, iTunes/Apple Music, Tidal, YouTube, Soulseek (slskd), Beatport, ListenBrainz, MusicBrainz, AcoustID, LRClib, music-map.com, Plex, Jellyfin, Navidrome
 
-**Stack**: Python 3.8+, Flask, SQLite, PyQt6 (desktop GUI in maintenance mode)
+**Stack**: Python 3.11, Flask, SQLite (WAL mode), vanilla JavaScript SPA
 
 **Core Components**:
-- Matching engine with Unicode/fuzzy logic
-- Discovery system with custom algorithms
-- Download pipeline with quality profiles
-- Metadata enhancement (lyrics, art, tags)
-- Template-based file organization
+- **Matching Engine** — version-aware fuzzy matching with multi-strategy query generation and source-specific confidence weighting
+- **Download Orchestrator** — routes between Soulseek and YouTube with hybrid fallback, batch processing with concurrent workers, automatic retry on failure/timeout
+- **Discovery System** — custom algorithms for personalized playlists, seasonal content, and similar artist exploration
+- **Metadata Pipeline** — dual-source (Spotify/iTunes) with MusicBrainz enrichment, AcoustID verification, LRC lyrics, album art embedding via mutagen
+- **Database Update Worker** — incremental sync from media servers with smart early-stopping (Jellyfin fast-path: ~2 API calls vs thousands)
+- **Web Scan Manager** — debounced media server scanning with completion callbacks
+- **Template-based File Organization** — configurable folder structures with automatic fallback
 
 ---
 
@@ -311,11 +368,18 @@ Transfer/Artist/Artist - Album/01 - Track.flac
 **Enable Debug Logging**: Settings → Log Level → DEBUG → Check `logs/app.log`
 
 **Common Issues**
-- **Files not organizing**: Verify transfer path, check template syntax, use "Reset to Defaults"
-- **Docker paths**: Ensure drives mounted in docker-compose.yml, use `/host/mnt/X/` prefix
-- **OAuth from remote**: Manually edit callback URL to server IP (Spotify requires 127.0.0.1)
-- **Wishlist stuck**: Auto-retry runs every 30 mins, check logs for failures
-- **Multi-library**: Select correct library in settings dropdown
+
+| Problem | Solution |
+|---------|----------|
+| Library not showing up | Run a Database Update from Dashboard (full refresh for first time) |
+| Files not organizing | Verify transfer path is writable, check template syntax, try "Reset to Defaults" |
+| Docker path issues | Ensure host paths are mapped in docker-compose.yml, use container paths in SoulSync settings |
+| slskd not connecting | Use `host.docker.internal` or your machine's LAN IP, not `localhost` |
+| OAuth from remote machine | Manually change `127.0.0.1` to your server's IP in the callback URL |
+| Wishlist not processing | Auto-retry runs on a timer, check logs for persistent failures |
+| Wrong track downloaded | Enable AcoustID verification in settings for fingerprint-based checking |
+| Post-processing file not found | Check that download and transfer paths are correctly mapped and writable |
+| Unraid permission errors | Set PUID/PGID to match your user (typically 99/100) |
 
 ---
 
@@ -330,10 +394,9 @@ Transfer/Artist/Artist - Album/01 - Track.flac
 - Mobile-responsive improvements
 
 ### Under Consideration
-- MusicBrainz ID integration
-- Additional streaming sources (Deezer, Apple Music)
+- Additional streaming sources (Deezer, Apple Music direct)
 - Playlist collaboration between instances
-- Machine learning for matching
+- Machine learning for matching improvement
 
 ---
 
@@ -345,7 +408,7 @@ MIT License - See [LICENSE](LICENSE) file for details
 
 ## Acknowledgments
 
-**Services**: slskd, music-map.com, LRClib.net, Spotify, Tidal, Plex, Jellyfin, Navidrome
+**Services**: slskd, music-map.com, LRClib.net, Spotify, iTunes, Tidal, YouTube, Plex, Jellyfin, Navidrome, MusicBrainz, AcoustID, Beatport
 
 **Community**: Contributors, testers, and users providing feedback