this post was submitted on 23 Nov 2023
3 points (100.0% liked)

Self-Hosted Main

515 readers
1 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

For Example

We welcome posts that include suggestions for good self-hosted alternatives to popular online services, how they are better, or how they give back control of your data. Also include hints and tips for less technical readers.

Useful Lists

founded 1 year ago
MODERATORS
communick@selfhosted.forum
3
rclone+Deluge+Plex: How do I automatically copy completed downloads? (alien.top)
submitted 11 months ago by agent-bagent@alien.top to c/main@selfhosted.forum
3 comments fedilink hide all child comments
 

I have a plex setup on Ubuntu with Deluge, Jackett, *arr apps, etc. All is gravy.

Except I currently need to manually run rclone copyto on completed Deluge torrents in order to [reliably] pipe the media to rclone/Plex.

Context:

  1. Deluge downloads to ~/local/
  2. rclone is mounted to ~/gdrive/ using:
rclone mount gdrive: ~/gdrive --allow-other --vfs-cache-mode off --bwlimit 15M --tpslimit 4 --tpslimit-burst 4
  1. Plex serves from ~/gdrive/Media/TV and ~/gdrive/Media/Movies

Initially I was using Deluge Labels to auto-move completed downloads, but this resulted in IO errors (and occasionally illegal disk seeks when I messed with enabling vfs cache writes).

I tried setting up a cron job to monitor ~/local/ for completed files then copying to rclone, but that resulted in IO errors.

The ONLY reliable solution I've found is manually running rclone copyto from a screen session for completed downloads. It works but it's a pain in the ass.

My questions for yall:

  1. Any idea why I'm getting these IO errors? I'm not pegging the CPU or the disk. There doesn't seem to be hardware issue, it really feels like the problem relates to rclone and Google's APIs

  2. Any idea how to troubleshoot this and get downloads automatically moving? Any logging mechanism I use (Deluge, rcloone, kernel) just gives generic IO errors. It doesn't point me in any direction

you are viewing a single comment's thread
view the rest of the comments
[โ€“] IsThisNameGoodEnough@alien.top 1 points 11 months ago (1 children)

I ran into the same problem when first setting up a similar program stack. The problem I found was that some programs need to have the rclone mount setup before they first attempt to access the local folders. Just allowing the programs to start randomly would typically result in I/O errors.

My solution was to use the healthcheck feature of docker-compose to ensure that the mount was accessible before anything downstream booted. I placed a small "test" file in the mount and wait until it's available. I use bash, which means the rclone container needs to have it installed. Below is the dockerfile I use for rclone and my docker-compose file that controls the boot-up sequence.

One thing to note: only docker-compose uses the depends-on argument, so when your server boots-up the default is that docker starts the containers at random. You can get around this by turning off the auto-start using docker and then setting a cron-job that runs the docker-compose up commands at start-up.

Rclone Dockerfile FROM rclone/rclone:latest RUN apk update &&
apk add bash

docker-compose.yml version: '3.5'

services:

## rclone
    rclone-plex-media:
        build: ./rclone/
        container_name: rclone-plex-media
        environment:
          - RCLONE_CONFIG=/config/rclone.conf
        cap_add:
          - SYS_ADMIN
        privileged: true
        devices:
          - /dev/fuse
        restart: unless-stopped
        volumes:
          - ./rclone/plex-media:/mnt/docker/media/rclone/plex-media-internal:shared
          - ./rclone/plex-cache:/cache
          - ./rclone/plex-config:/config
        command: mount "plex-encrypted:"  "/mnt/docker/media/rclone/plex-media-internal" --uid 1000 --gid 1000 --umask 002 --default-permissions --allow-non-empty --allow-other --vfs-cache-mode full --cache-dir "/cache/" --vfs-cache-max-size 50G --dir-cache-time 1000h --vfs-cache-max-age 9999h --vfs-write-back 24h --log-level INFO --poll-interval=15s --stats 1m
        healthcheck:
          test: bash -c "[ -f /mnt/docker/media/rclone/plex-media-internal/test.txt ]"
          interval: 30s
          retries: 3
          start_period: 30s
          timeout: 5s
            
## Streaming - Plex
    plex:
        image: plexinc/pms-docker
        container_name: plex
        environment:
            - TZ=America/New_York
        devices:
            - /dev/dri:/dev/dri
        restart: unless-stopped
        network_mode: host
        volumes:
          - ./plex/config:/config
          - ./plex/transcode:/transcode
          - ./rclone/plex-media:/data
        depends_on:
          rclone-plex-media:
            condition: service_healthy
            
## Plex Scan Trigger - Autoscan
    autoscan:
        image: cloudb0x/autoscan
        container_name: autoscan
        environment:
          - PUID=1000
          - PGID=1000
        volumes:
          - ./autoscan/config:/config
          - ./rclone/plex-media:/mnt/unionfs:ro
        ports:
          - 3030:3030
        restart: unless-stopped
        depends_on:
          plex:
            condition: service_healthy        

## Media Management - Deluge
    deluge:
        image: binhex/arch-delugevpn
        container_name: deluge
        privileged: true
        sysctls:
          - net.ipv4.conf.all.src_valid_mark=1
        volumes:
          - ./deluge/downloads:/downloads
          - ./deluge/config:/config
          - /etc/localtime:/etc/localtime:ro
        environment:
          - VPN_ENABLED=yes
          - VPN_USER=REMOVED
          - VPN_PASS=REMOVED
          - VPN_PROV=REMOVED
          - VPN_CLIENT=wireguard
          - STRICT_PORT_FORWARD=yes
          - ENABLE_PRIVOXY=yes
          - LAN_NETWORK=192.168.2.0/24
          - NAME_SERVERS=REMOVED
          - DELUGE_DAEMON_LOG_LEVEL=info
          - DELUGE_WEB_LOG_LEVEL=info
          - VPN_INPUT_PORTS=1234
          - VPN_OUTPUT_PORTS=5678
          - DEBUG=false
          - UMASK=000
          - PUID=1000
          - PGID=1000
        ports:
          - 8112:8112
          - 8118:8118
          - 58846:58846
          - 58946:58946
        healthcheck:
          test: "curl -f http://localhost:8112 || exit 1"
          interval: 10s
          retries: 3
          start_period: 600s
          timeout: 5s
        restart: unless-stopped

## Media Management - Prowlarr
    prowlarr:
        image: lscr.io/linuxserver/prowlarr:develop
        container_name: prowlarr
        environment:
          - PUID=1000
          - PGID=1000
          - TZ=America/New_York
        volumes:
          - ./prowlarr/config:/config
        ports:
          - 9696:9696
        restart: unless-stopped
        healthcheck:
          test: "curl -f http://localhost:9696 || exit 1"
          interval: 10s
          retries: 6
          start_period: 5s
          timeout: 5s
        depends_on:
          rclone-plex-media:
            condition: service_healthy
          deluge:
            condition: service_healthy
    
## Media Management - Radarr
    radarr:
        image: lscr.io/linuxserver/radarr:latest
        container_name: radarr
        environment:
          - PUID=1000
          - PGID=1000
          - TZ=America/New_York
        volumes:
          - ./radarr/config:/config
          - ./rclone/plex-media/Movies:/movies
          - ./deluge/downloads:/downloads
        ports:
          - 7878:7878
        restart: unless-stopped
        healthcheck:
          test: "curl -f http://localhost:7878 || exit 1"
          interval: 10s
          retries: 6
          start_period: 5s
          timeout: 5s
        depends_on:
          rclone-plex-media:
            condition: service_healthy
          prowlarr:
            condition: service_healthy

## Media Management - Sonarr
    sonarr:
        image: lscr.io/linuxserver/sonarr:latest
        container_name: sonarr
        environment:
          - PUID=1000
          - PGID=1000
          - TZ=America/New_York
        volumes:
          - ./sonarr/config:/config
          - './rclone/plex-media/TV Shows:/tv'
          - ./deluge/downloads:/downloads
        ports:
          - 8989:8989
        restart: unless-stopped
        healthcheck:
          test: "curl -f http://localhost:8989 || exit 1"
          interval: 10s
          retries: 6
          start_period: 5s
          timeout: 5s
        depends_on:
          rclone-plex-media:
            condition: service_healthy
          prowlarr:
            condition: service_healthy
[โ€“] IsThisNameGoodEnough@alien.top 1 points 11 months ago

One more thing. I'd highly recommend using vfs-cache for rclone since it takes care of moving the files to your remote source. You can see in my setup rclone will keep a local copy for 24 hrs and then automatically transfer it to the remote.

I was originally using an external script to copy/move files but vfs-cache is so much easier. Just let rclone handle everything in the background.