diff --git a/README.md b/README.md index ebaeee3a..f4f679f6 100644 --- a/README.md +++ b/README.md @@ -2,53 +2,161 @@ SoulSync Logo

-# 🎡 SoulSync - Automated Music Discovery & Collection Manager +# SoulSync - Intelligent Music Discovery & Automation Platform -**Bridge streaming services to your local music library.** Automatically sync Spotify/Tidal/YouTube playlists to Plex/Jellyfin/Navidrome via Soulseek with intelligent matching, metadata enhancement, and automated discovery. +**Bring Spotify-quality music discovery to your self-hosted library.** SoulSync automates music collection with intelligent discovery algorithms, multi-source downloads, and zero manual intervention. -> ⚠️ **CRITICAL**: Configure file sharing in slskd before use. Users who only download without sharing get banned by the Soulseek community. Set up shared folders at `http://localhost:5030/shares`. +> **IMPORTANT**: Configure file sharing in slskd before use. The Soulseek community bans users who only download without sharing. Set up shared folders at `http://localhost:5030/shares`. -> πŸ“’ **Development Focus**: New features are developed for the **Web UI** version. The Desktop GUI receives maintenance and bug fixes only. +> **Development Status**: New features are developed for the **Web UI**. The Desktop GUI receives maintenance and bug fixes only. -## πŸ’¬ Community +**Community**: [Discord Server](https://discord.gg/ePx7xYuV) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad) -Join the Discord server for support, feature requests, and discussions: -- **Discord**: [https://discord.gg/ePx7xYuV](https://discord.gg/ePx7xYuV) +--- + +## The Problem -## ✨ Core Features +You want Spotify's discovery features (Release Radar, Discovery Weekly, personalized playlists) but for your local music library. Existing tools require manual playlist management or lack intelligent discovery. -**Search & Download** -- **Enhanced Search**: Unified search across Spotify, your library, and Soulseek with categorized results (artists, albums, tracks) -- **Basic Search**: Direct Soulseek search with instant streaming and download -- Auto-sync playlists from Spotify/Tidal/YouTube to your media server -- Smart matching against your existing library -- FLAC-priority downloads from Soulseek with automatic fallback -- Customizable file organization with template-based path structures -- Synchronized lyrics (LRC) for every track via LRClib.net +## The Solution -**Metadata & Organization** -- Enhanced metadata with album art and proper tags -- Flexible folder templates: `$albumartist/$album/$track - $title` -- Automatic library scanning and database updates -- Clean, organized music collection +SoulSync bridges streaming services to your self-hosted media server with **automated discovery and collection**: -**Discovery & Automation** -- Browse complete artist discographies with similar artist recommendations -- Intelligent music discovery using your watchlist ([music-map.com](https://music-map.com) integration) -- Curated playlists: Release Radar, Discovery Weekly, Seasonal Mixes -- Beatport chart integration for electronic music -- Artist watchlist monitors new releases automatically +1. **Monitors artists** for new releases automatically +2. **Generates personalized playlists** using custom recommendation algorithms +3. **Downloads missing tracks** from Soulseek with FLAC priority +4. **Enriches metadata** with lyrics, album art, and proper tags +5. **Organizes files** using customizable templates +6. **Syncs with Plex/Jellyfin/Navidrome** automatically + +--- -**Management** -- Comprehensive library browser with search and completion tracking -- Wishlist system with automatic retry (30-minute intervals) -- Granular wishlist management (remove individual tracks or entire albums) -- Dynamic log level control (DEBUG/INFO/WARNING/ERROR) -- Background automation handles retries and database updates +## Key Features + +### Intelligent Discovery System + +**Release Radar** - 30 new tracks from your watchlist, updated daily +- Monitors 100+ artists automatically +- Only includes releases from the last 30 days +- Curated using popularity scoring + +**Discovery Weekly** - 50 tracks from similar artists you don't own +- Custom algorithm: 20 popular + 20 mid-tier + 10 deep cuts +- Built from 1000+ track discovery pool +- Updated every 24 hours + +**Seasonal Playlists** - Auto-generated themed collections +- Halloween, Christmas, Valentine's Day, Summer, Spring, Autumn +- Smart keyword matching and genre analysis +- Active during appropriate months + +**Personalized Playlists** (12+ types) +- Recently Added, Top Tracks, Forgotten Favorites +- Decade Playlists (1960s-2020s) +- Genre Playlists (15 parent categories, 50+ sub-genres) +- Daily Mixes, Hidden Gems, Popular Picks +- Custom Playlist Builder (seed with artists) + +**ListenBrainz Integration** +- Import recommendation playlists +- Sync user and collaborative playlists +- Access community-curated content + +**Beatport Integration** - Electronic music discovery +- Browse by genre (House, Techno, Trance, Drum & Bass, etc.) +- Top 100, Hype Charts, DJ Charts +- Staff Picks, New Releases, Latest Tracks +- Per-genre hero sections and featured content + +### Multi-Source Downloads + +**Primary Sources** +- **Soulseek**: FLAC-priority with automatic quality selection +- **Beatport Charts**: Electronic music with Spotify matching +- **Spotify Playlists**: Public and private playlist sync +- **Tidal Playlists**: Alternative streaming source +- **YouTube Playlists**: Fallback option + +**Smart Download Pipeline** +- Quality profiles: Audiophile, Balanced, Mobile +- Automatic format fallback (FLAC β†’ MP3 β†’ other) +- Duplicate prevention against existing library +- Batch processing with concurrent downloads +- Automatic retry on failure (30-minute intervals) + +### Advanced Matching Engine + +**Text Normalization** +- Unicode handling (handles KoΠ―n, BjΓΆrk, etc.) +- Special character preservation (A$AP Rocky) +- Accent normalization (BeyoncΓ© β†’ Beyonce) +- Abbreviation expansion (feat. β†’ featured, pt. β†’ part) + +**Fuzzy Matching** +- Multi-strategy: exact β†’ normalized β†’ Unicode fallback +- Album variation handling (Deluxe, Remastered, Platinum Edition) +- Artist name preservation (doesn't break "Daryl Hall & John Oates") +- Confidence scoring with configurable thresholds + +### Automation & Monitoring + +**Watchlist System** +- Monitor unlimited artists for new releases +- Automatic similar artist discovery via music-map.com +- Occurrence-based ranking (tracks artist overlap) +- Configurable scan intervals + +**Wishlist System** +- Tracks failed downloads automatically +- Auto-retry every 30 minutes +- Granular management (remove tracks or entire albums) +- Source tracking (playlist, album, manual) + +**Background Tasks** +- Database synchronization with media server +- Discovery pool population (50 artists Γ— 10 releases) +- Seasonal content updates +- Library completion tracking + +### Library Management + +**Tools** +- **Quality Scanner**: Find low-bitrate files to replace +- **Duplicate Cleaner**: Identify redundant tracks +- **Completion Tracking**: See album progress percentages +- **Enhanced Search**: Unified search across Spotify, library, and Soulseek + +**Metadata Enhancement** +- Synchronized lyrics (LRC format) via LRClib.net +- Album art embedding +- Proper ID3/Vorbis tags +- Custom file organization templates + +**File Organization** +- Template-based paths: `$albumartist/$album/$track - $title` +- Separate templates for albums, singles, playlists +- Client-side validation +- Automatic fallback on errors + +### Media Server Integration + +**Supported Servers** +- Plex (with library selection) +- Jellyfin (with multi-library support) +- Navidrome -## πŸš€ Installation +**Features** +- Automatic library scanning after downloads +- Database caching for fast access +- Incremental updates +- Connection testing and validation + +--- + +## Installation ### Docker (Recommended) + ```bash # Using docker-compose curl -O https://raw.githubusercontent.com/Nezreka/SoulSync/main/docker-compose.yml @@ -60,7 +168,8 @@ docker run -d -p 8008:8008 boulderbadgedad/soulsync:latest # Access at http://localhost:8008 ``` -### Web UI (Python) +### Python (Web UI) + ```bash git clone https://github.com/Nezreka/SoulSync cd SoulSync @@ -70,6 +179,7 @@ python web_server.py ``` ### Desktop GUI + ```bash git clone https://github.com/Nezreka/SoulSync cd SoulSync @@ -77,58 +187,222 @@ pip install -r requirements.txt python main.py ``` -## ⚑ Quick Setup +--- + +## Quick Setup ### Prerequisites -- **slskd**: [Download](https://github.com/slskd/slskd/releases), run on port 5030 -- **Spotify API**: Client ID/Secret from [Developer Dashboard](https://developer.spotify.com/dashboard) -- **Tidal API** (optional): Client ID/Secret from [Developer Dashboard](https://developer.tidal.com/dashboard) + +- **slskd** running on port 5030 ([Download](https://github.com/slskd/slskd/releases)) +- **Spotify API** credentials ([Developer Dashboard](https://developer.spotify.com/dashboard)) +- **Tidal API** (optional) ([Developer Dashboard](https://developer.tidal.com/dashboard)) - **Media Server** (optional): Plex, Jellyfin, or Navidrome -### API Credentials +### API Configuration **Spotify** -1. [Create app](https://developer.spotify.com/dashboard) β†’ Settings +1. Create app at [Developer Dashboard](https://developer.spotify.com/dashboard) 2. Add redirect URI: `http://127.0.0.1:8888/callback` 3. Copy Client ID and Secret -**Tidal** -1. [Create app](https://developer.tidal.com/dashboard) +**Tidal** (optional) +1. Create app at [Developer Dashboard](https://developer.tidal.com/dashboard) 2. Add redirect URI: `http://127.0.0.1:8889/callback` 3. Add scopes: `user.read`, `playlists.read` 4. Copy Client ID and Secret -**Plex** -- Get token from any media item URL: `?X-Plex-Token=YOUR_TOKEN` +**Plex** (optional) +- Get token from media item URL: `?X-Plex-Token=YOUR_TOKEN` - Server URL: `http://YOUR_IP:32400` -**Jellyfin** +**Jellyfin** (optional) - Settings β†’ API Keys β†’ Generate new key - Server URL: `http://YOUR_IP:8096` -**Navidrome** +**Navidrome** (optional) - Settings β†’ Users β†’ Generate API Token -- Or use username/password - Server URL: `http://YOUR_IP:4533` -### Configuration +### Initial Configuration + +1. Launch SoulSync and navigate to Settings +2. Enter API credentials for streaming services +3. Configure media server connection (if using) +4. Set slskd URL (`http://localhost:5030`) and API key +5. Configure download path and transfer path +6. Customize file organization templates (optional) +7. **Configure file sharing in slskd to avoid bans** + +### Docker-Specific Setup + +**Path Mapping** +```yaml +volumes: + - ./config:/app/config # Settings persist + - ./logs:/app/logs # Log files + - /mnt/c:/host/mnt/c:rw # Mount Windows drives + - /mnt/d:/host/mnt/d:rw +``` + +Use `/host/mnt/X/path` in settings where X is your drive letter. + +**OAuth Authentication from Remote Devices** + +Due to Spotify API requirements (127.0.0.1 mandatory, localhost banned), remote OAuth needs a workaround: + +1. Complete OAuth flow - redirected to `http://127.0.0.1:8888/callback?code=...` +2. Manually edit URL to your server IP: `http://192.168.1.5:8888/callback?code=...` +3. Press Enter to complete authentication + +See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md) for details. + +--- + +## How It Works + +### Discovery Pipeline + +1. **Add artists to watchlist** β†’ SoulSync monitors for new releases +2. **Fetch similar artists** β†’ music-map.com provides 10 similar artists per watchlist artist +3. **Aggregate by occurrence** β†’ Ranks similar artists by how many watchlist artists recommend them +4. **Build discovery pool** β†’ Top 50 similar artists Γ— 10 recent releases = ~500 albums +5. **Extract tracks** β†’ Pool contains 1000-2000 tracks, rolling 1-year window +6. **Curate playlists** β†’ Algorithms generate Release Radar, Discovery Weekly, Seasonal + +### Download Workflow + +1. **Source Selection** β†’ User picks playlist, album, or uses discovery features +2. **Library Matching** β†’ SoulSync checks existing library to avoid duplicates +3. **Quality Filtering** β†’ Applies user-defined quality profile (FLAC priority) +4. **Download Queue** β†’ Batches requests with configurable concurrency (default: 3) +5. **Metadata Enhancement** β†’ Adds lyrics (LRC), album art, proper tags +6. **File Organization** β†’ Moves to transfer folder using custom templates +7. **Media Server Sync** β†’ Triggers library rescan, updates internal database + +### Automation Loop + +- **Every 30 minutes**: Wishlist retry for failed downloads +- **Every 24 hours**: Discovery pool refresh, playlist curation +- **On media server scan**: Database incremental update +- **On new release**: Watchlist triggers download if configured + +--- + +## Who Should Use SoulSync + +### Perfect For + +- **Self-hosters** with Plex, Jellyfin, or Navidrome libraries +- **Music enthusiasts** with 500+ album collections who want automated discovery +- **Electronic music fans** who follow Beatport charts +- **Former Spotify users** who want discovery features for local files +- **Power users** comfortable with API configuration and Docker + +### Not Ideal For + +- Casual users wanting simple one-click playlist sync +- Users on slow/metered internet (download-heavy workflow) +- People uncomfortable with terminal commands or API keys +- Those seeking streaming-only solutions (not a media server replacement) + +--- + +## Comparison with Alternatives + +| Feature | SoulSync | Lidarr | Headphones | Beets | +|---------|----------|--------|------------|-------| +| **Custom Discovery Algorithm** | βœ“ | βœ— | βœ— | βœ— | +| **Personalized Playlists** | 12+ types | Manual lists | βœ— | βœ— | +| **Beatport Integration** | βœ“ (charts, genres) | βœ— | βœ— | βœ— | +| **ListenBrainz Playlists** | βœ“ | βœ— | βœ— | βœ— | +| **Multi-Source Downloads** | Spotify/Tidal/YouTube | MusicBrainz | βœ— | βœ— | +| **Watchlist Monitoring** | βœ“ (100+ artists) | βœ“ (manual add) | βœ“ | βœ— | +| **LRC Lyrics** | βœ“ (auto) | βœ— | βœ— | Plugin | +| **Advanced Matching** | Unicode, fuzzy, confidence | Basic | Basic | βœ“ | +| **Quality Scanner** | βœ“ | βœ— | βœ— | βœ“ | +| **Duplicate Cleaner** | βœ“ | βœ— | βœ— | βœ“ | +| **Web UI** | Modern Flask | βœ“ | Basic | CLI only | +| **Template-Based Organization** | βœ“ | βœ— | βœ— | βœ“ | +| **Seasonal Playlists** | βœ“ (auto) | βœ— | βœ— | βœ— | + +**SoulSync's Unique Position**: Only tool combining intelligent discovery (Release Radar, Discovery Weekly) with multi-source automation (Beatport charts, ListenBrainz) and self-hosted library management. + +--- + +## Architecture + +### Technical Stack + +- **Language**: Python 3.8+ +- **Web Framework**: Flask with 120+ API endpoints +- **Database**: SQLite with connection pooling and indexing +- **UI**: Modern JavaScript with real-time updates +- **Desktop**: PyQt6 (maintenance mode) + +### Service Integrations + +- **Spotify API**: Artist monitoring, playlist sync, metadata +- **Tidal API**: Alternative playlist source +- **Plex API**: Library scanning, metadata sync +- **Jellyfin API**: Multi-library support +- **Navidrome API**: Subsonic-compatible server +- **Slskd API**: Download management, search +- **ListenBrainz API**: Community playlists, recommendations +- **LRClib.net**: Synchronized lyrics +- **music-map.com**: Similar artist discovery + +### Core Components + +**Matching Engine** (`core/matching_engine.py`) +- Text normalization with Unicode support +- Fuzzy string matching with confidence scoring +- Album variation handling +- Special character preservation + +**Discovery System** (`core/watchlist_scanner.py`, `core/personalized_playlists.py`) +- Watchlist monitoring +- Discovery pool population +- Playlist curation algorithms +- Seasonal content generation + +**Download Pipeline** (`core/soulseek_client.py`, `services/sync_service.py`) +- Quality profile filtering +- Concurrent download management +- Automatic retry logic +- Batch processing + +**Metadata Enhancement** (`core/lyrics_client.py`) +- LRC lyrics fetching +- Album art embedding +- Tag normalization +- File organization + +### Database Schema + +- **Tracks**: Full track metadata with file paths +- **Albums**: Album info with completion tracking +- **Artists**: Artist profiles with watchlist status +- **Discovery Pool**: 1000-2000 track rotating pool +- **Seasonal Content**: Cached seasonal albums/tracks +- **Wishlist**: Failed downloads with retry tracking +- **Similar Artists**: Occurrence-ranked recommendations + +### Scale + +- **83,000+ lines** of Python code +- **120+ API endpoints** +- **15+ service clients** +- **20+ database tables** +- **Handles libraries of 10,000+ albums** + +--- -1. Launch SoulSync and go to Settings -2. Enter API credentials for streaming services and media server -3. Configure slskd URL (`http://localhost:5030`) and API key -4. Set download and transfer paths -5. **Customize file organization** (optional): - - Enable custom templates in Settings β†’ File Organization - - Default: `$albumartist/$albumartist - $album/$track - $title` - - Variables: `$artist`, `$albumartist`, `$album`, `$title`, `$track`, `$playlist` - - Example: `Music/$artist/$year - $album/$track - $title` -6. **Share files in slskd** to avoid bans +## File Organization -## πŸ“ File Organization +SoulSync uses customizable path templates with validation. -SoulSync supports customizable path templates with validation and fallback protection. +### Default Structure -**Default Structure** ``` Transfer/ Artist/ @@ -137,98 +411,143 @@ Transfer/ 01 - Track.lrc ``` -**Template System** +### Template System + +**Available Variables** +- `$artist` - Track artist +- `$albumartist` - Album artist +- `$album` - Album name +- `$title` - Track title +- `$track` - Track number (zero-padded: 01, 02...) +- `$playlist` - Playlist name + +**Default Templates** - **Albums**: `$albumartist/$albumartist - $album/$track - $title` - **Singles**: `$artist/$artist - $title/$title` - **Playlists**: `$playlist/$artist - $title` -**Available Variables** -- `$artist`, `$albumartist`, `$album`, `$title` -- `$track` (zero-padded: 01, 02...) -- `$playlist` (playlist name) - **Features** - Client-side validation prevents invalid templates -- Reset to defaults button in settings -- Automatic fallback if template fails -- Changes apply immediately to new downloads - -## 🐳 Docker Notes +- Automatic fallback on errors +- Reset to defaults button +- Changes apply immediately -**Path Configuration** -```yaml -volumes: - - ./config:/app/config # Settings persist - - ./logs:/app/logs # Log files - - /mnt/c:/host/mnt/c:rw # Mount Windows drives - - /mnt/d:/host/mnt/d:rw -``` - -Use `/host/mnt/X/path` in settings where X is your drive letter. - -**OAuth from Remote Devices** -When accessing from a different machine, Spotify redirects may fail: -1. Complete OAuth flow - get redirected to `http://127.0.0.1:8888/callback?code=...` -2. Edit URL to use your server IP: `http://192.168.1.5:8888/callback?code=...` -3. Press Enter to complete authentication - -See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md) for details. - -## πŸ“Š Workflow +--- -1. **Sync**: Select Spotify/Tidal/YouTube playlist -2. **Match**: SoulSync compares against your library -3. **Download**: Missing tracks queued from Soulseek -4. **Process**: Files enhanced with metadata, lyrics, and album art -5. **Organize**: Moved to transfer folder with template-based structure -6. **Scan**: Media server automatically rescans library -7. **Update**: SoulSync database syncs with your collection +## Troubleshooting -## πŸ› Troubleshooting +### Enable Debug Logging -**Enable Debug Logging** -- Settings β†’ Log Level β†’ DEBUG -- Check `logs/app.log` for detailed information -- Change takes effect immediately +Settings β†’ Log Level β†’ DEBUG (takes effect immediately) +Check `logs/app.log` for detailed information -**Common Issues** +### Common Issues -*Files not organizing properly* +**Files not organizing properly** - Verify transfer path points to your music library - Check template syntax in Settings β†’ File Organization - Use "Reset to Defaults" if templates are broken - Review logs for path-related errors -*Docker drive access* +**Docker drive access issues** - Ensure drives are mounted in docker-compose.yml - Restart Docker Desktop if mounts fail -- Verify paths use `/host/mnt/X/` prefix +- Verify paths use `/host/mnt/X/` prefix in settings + +**OAuth failing from remote devices** +- Spotify requires 127.0.0.1, not server IP +- Manually edit callback URL to use server IP +- See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md) -*Wishlist tracks stuck* +**Wishlist tracks stuck** - Remove items using delete buttons on wishlist page - Auto-retry runs every 30 minutes -- Check logs for download failures +- Check logs for persistent download failures +- Verify slskd is running and accessible -*Multi-library setups* -- Select correct library from dropdown in settings (Plex/Jellyfin) +**Multi-library Plex/Jellyfin setups** +- Select correct library from dropdown in settings - Test connection to verify credentials +- Check library permissions + +**Quality scanner finding false positives** +- Adjust quality profile thresholds +- Review format priorities (FLAC vs. MP3) +- Check logs for matching errors + +--- + +## Development + +### Project Structure + +``` +SoulSync/ +β”œβ”€β”€ core/ # Core service clients +β”‚ β”œβ”€β”€ spotify_client.py +β”‚ β”œβ”€β”€ soulseek_client.py +β”‚ β”œβ”€β”€ matching_engine.py +β”‚ β”œβ”€β”€ watchlist_scanner.py +β”‚ └── personalized_playlists.py +β”œβ”€β”€ database/ # Database layer +β”‚ └── music_database.py +β”œβ”€β”€ services/ # Business logic +β”‚ └── sync_service.py +β”œβ”€β”€ webui/ # Web interface +β”‚ β”œβ”€β”€ static/ +β”‚ └── index.html +β”œβ”€β”€ ui/ # Desktop GUI (PyQt6) +β”œβ”€β”€ config/ # Configuration management +β”œβ”€β”€ utils/ # Utilities and logging +└── web_server.py # Flask application (22k lines) +``` + +### Contributing -## πŸ—οΈ Architecture +Contributions welcome! Please: +1. Check existing issues before creating new ones +2. Follow existing code style +3. Add tests for new features +4. Update documentation -- **Services**: Spotify, Tidal, Plex, Jellyfin, Navidrome, Soulseek clients -- **Database**: SQLite with automatic library caching and updates -- **UI**: PyQt6 Desktop + Flask Web Interface -- **Matching**: Advanced text normalization and fuzzy scoring -- **Metadata**: Mutagen + LRClib.net for tags and lyrics -- **Automation**: Multi-threaded with retry logic and background tasks +--- + +## Roadmap + +### Planned Features + +- WebSocket support (replace polling architecture) +- Batch wishlist operations (select 20 tracks β†’ remove) +- Download history browser UI +- Source reliability tracking (learn which Slskd users are best) +- Notification center (persistent toast history) +- Mobile-responsive UI improvements +- Playlist collaboration between SoulSync instances +- Smart bandwidth management (time-based rules) + +### Under Consideration + +- MusicBrainz ID integration +- Additional streaming sources (Deezer, Apple Music) +- Advanced playlist scheduling +- Export to external playlists (Spotify, Tidal) +- Machine learning for improved matching + +--- + +## License + +[Include your license here] + +--- -## πŸ“ Recent Updates +## Acknowledgments -- **Customizable file organization** with template-based paths and validation -- **Log level control** without restart -- **Jellyfin library selector** for multi-library setups -- **Enhanced wishlist management** with track/album removal -- **Docker config persistence** between container restarts +- **slskd** - Soulseek daemon +- **music-map.com** - Similar artist data +- **LRClib.net** - Synchronized lyrics +- **Spotify, Tidal, Plex, Jellyfin, Navidrome** - API providers +- **Community contributors** - Feature requests and bug reports ---