Hard Earned Lessons When Doing Backups on Windows Using Robocopy

This afternoon, I tried updating the set of secondary backup drives I'd previously spent Easter weekend painstakingly (and very frustratingly) spent waiting several *DAYS* to complete...

At the time, I'd ended up doing a bunch of reading, about what may have been going wrong (especially as it seemed awfully slow + hard to predict what was going on / when the copy jobs "may" complete). Well, turns out I needed to call on those lessons today, confirming that that does seem to have been the culprit  (i.e. it had only made progress over like 2-3 folders for the past month's data, when usually each time I'd copied those using Windows Explorer onto another drive, the process was usually way quicker!)

In Short:

It seems that that combination of "robocopy /mt" and "exFAT" drives in particular is slower than a frozen snail!🤬

So yeah, don't do what I did over Easter Weekend  (contributing to over half of my tech pain that weekend):

* 1) Do NOT use "robocopy /mt" when copying to an external HDD - Apparently, the "/mt" arg only really makes sense if you're copying between local drives, and really, only when one of those is a SSD even. (If it's over USB, or to anything other than NTFS... yeah, don't bother!)

* 2) Especially do NOT use "robocopy /mt" when copying to an "exFAT" formatted external hard drive

* 3) Probably do NOT use Window's Explorer's "Format Drive" right click menu to do so either!  (With my other external exFAT HDD drive, I had to get it to do a "scan + fix errors" on it after I aborted the first round of copies!) 

* 4) Oh, and make sure you've got something like "wakepy" running when you start up your copy job, so that your machine doesn't end up going to sleep during the copy operation before it finishes (resulting in a job you thought you left unattended to run as-quick-as-possible overnight actually terminating about 2-4 hours after you left the room) - This last tip applies to both Windows AND Linux...  Both of these apparently ignore the fact that robocopy or rsync are busily chugging away:  Robocopy doesn't set the necessary flag that wakepy does, while rsync on Linux err... I can't remember the details now!   (e.g. See my "wakepy_keepalive.py" script in my "cmdutils" repo)

 

Outcome: 

I've now replaced corresponding flags for the other drive's backup script too, so hopefully that will  faster when I get there!  (There's still half a month's data to copy at time of writing this post)

 

NOTE: The problem isn't as pronounced when the drives were formatted as NTFS, hence I hadn't really bothered. But as exFAT... yeah... big mistake!  (It's too late to reformat those drives now and re-copy them from scratch again without having to waste another weekend. I'm better off making a third set of backups instead, to be able to have rotating backups instead!)