$ sudo reboot_

// Homelab

Smuggling Google Drive home, one nightly rclone run at a time

by Jon · · 2 min read

The platter and read arm inside an opened hard disk drive
Photo: Mk2010, CC BY 4.0, via Wikimedia Commons

The plan sounded simple: everything in Google Drive should also live on my own server, inside a self-hosted file cloud, without deleting a single thing on Google's side yet. rclone is the obvious tool for cloud-to-cloud copying, and it did the heavy lifting. It just took three separate surprises to get a clean run.

The setup

  • A Google Drive remote with a read-only scope. The job can't change anything in Google Drive even if I fat-finger a command.
  • A WebDAV remote for the self-hosted cloud, signed in with an app token instead of a real password.
  • A nightly scheduled job that pings my uptime monitor when it finishes, so I hear about it when it doesn't run.

Surprise #1: Google Docs aren't files

Native Google Docs, Sheets and Slides don't actually exist as files. rclone exports them on the fly (to docx, xlsx, pptx), which means their size is unknown until the export is done. My WebDAV server flatly refused uploads of unknown size, so every Google Doc failed while everything else sailed through. The fix was to stop going direct and add a pit stop:

# 1. Google Drive -> local staging folder (exports Google Docs)
rclone copy gdrive: /staging --drive-export-formats docx,xlsx,pptx,pdf
# 2. staging -> self-hosted cloud (every file now has a real size)
rclone copy /staging cloud:"Google Drive"

Surprise #2: the clever upload mode wasn't

rclone's server-specific WebDAV mode uses a resumable upload protocol, and it kept dying with an EOF partway through big files. Switching the remote to a more generic WebDAV vendor setting gave me plain, boring, reliable uploads.

Surprise #3: the 100 MB wall

The site sits behind a CDN that caps any single request at 100 MB. Large uploads from the web interface kept failing until I set the server's upload chunk size to 50 MB, so no single request ever touches the limit.

Leaving things behind on purpose

Photos already have a dedicated photo app, so that folder is excluded. While checking the first copy I also found a couple of old exports with credentials in them, which have no business sitting in a file share. They're excluded from the copy now, and the ones that had already landed got moved aside.

Before any migration, search the source for files like *passwords*.csv. I promise you'll find something.