True Story About Backing Up Data

I can not stress enough how important backups are, especially if you use your computer equipment to make a living. You can:
1- lose data and not get a paycheck from a client cause the work won’t get finished,
or more scary,
2- lose everything you own when the client sues you for data loss, a very serious libel suit because you are in fact legally responsible for keeping the clients’ data safe!

There is information to get help with a backup system and routine maintenance to reduce your data lose risk at the end of this blog post.

This is a true story, and the TV station was super fortunate to not have lost everything. But next time, they will, they’ve been warned, they’ll learn the hard way.


What You Should NOT Do

As of this writing, I’m contracted part-time to a local TV station, running a small production dept. Small low budget place, but enjoyable, and a paycheck until something better comes along. I switched them from a PPro shop to a strictly FCPX shop. Turnaround times are a fraction of what they used to be. They had me working on a mid-2010 Mac Pro, max factory specs. All media and Libraries were on an 8 bay “HDOne” RAID device. This was something introduced at 2008 NAB Show if my information is correct.

One day the system up and quits working correctly, out of the blue, no symptoms, no warnings. Can’t move data, constant freezes. Long story short, both the Mac Pro AND the RAID went bad AT THE SAME TIME! This station is on a shoe-string-budget so there’s no backups of anything anywhere. I do run Disk Utility to repair both drives weekly, and Disk Warrior monthly. Plus keep an eye on how much free space is left on both, which was plenty. But don’t blame this on old hardware, because brand new hardware goes out just the same way, just as often. In IT, the age of something doesn’t determine its health or stability, just its performance levels.

The Mac Pro started to either shut down or reboot itself randomly here and there. I’d say reset the PRAM and replace the system drive and see what happens, but we don’t have that sort of time or money. They’re just calling it dead and purchased a used iMac 5K (2015 model) off of eBay. Much faster than the old Mac Pro, like it a lot. BTW, I use a 2013 Mac Pro “tube” at home, maxed out, and this iMac isn’t as fast as it. I’ve done tons of tests. We’re also now using a brand new OWC 10TB dual drive RAID 0 system. Yep, if one drive goes, it’s all gone. But it was cheap and had the TB’s we need.

The HDOne RAID 5 I found has a flakey controller board. The drives all test fine with Disk Utility and Disk Warrior. I’ve got an extensive IT background, I’m very familiar with this stuff. The controller board is bad, unstable, so not like we can swap out a drive and presto-changer it’s fixed. And it won’t transfer large files, so there’s stuff I can’t get off of it, but fortunately I got enough to make life easier. BUT, there’s a ton of stuff lost forever, because there’s no backup. AND I spend a whole 2 weeks trying to get this stuff off, putting me behind on other production work I now have to catch up on quickly. So life the past 2 weeks has been stressful.

Lots of FCPX plugins we’re having to contact developers and get licenses reset, recovered, etc. And there were a LOT. FxFactory and MotionVFX were the two easy ones, just fire up their stand alone app, sign in, done. And thanks to Apple’s App Store we could do the same for 80% of our daily production apps. Slowly coming back, it is a royal PITA, but, we didn’t lose everything, and we’re super lucky about that. We did in fact lose some data that will NEVER be recovered. Archived stuff that will be missed, some occasionally used stuff that will have to be recreated from scratch when necessary.

And the station still won’t pay for backup drives. But hey, I’m just a contractor, I put in my 2 cents, if things go bad, it’s not on me. And I will definitely say “I Todaso” (my favorite Ricky-ism).


What You SHOULD Do

A good backup system is very simple, it only takes two things: 1- a backup drive of equal capacity to the working drive, 2- reliable backup software that can work on an automated schedule.

Before I say anything else, Apple’s TimeMachine works for small 1 or 2 man shops that don’t have massive workflows or broadcast fast turnarounds. It does have its limits, and recovering may require jumping through some hoops. I personally am NOT a fan of it for the working professional. I don’t use it, I don’t bother with it, I don’t support it.

On my home edit suit, I have a Promise P2 RAID (Thunderbolt) and it runs great. I have a cheap OWC USB RAID to match the capacity and use it as my backup. Backups don’t need to be fast, just there. Hell, get a USB 1 drive for your backups, it doesn’t matter, as it happens when you’re not working, and just needs to exist, not be fast or anything else.

I use Carbon Copy Cloner to run my backups. They’re scheduled for 1am every morning as an appended backup. That means CCC only copies files not already on the backup drive. And deletes files not on my working P2 RAID. I have a perfect mirror every night at 1am. The initial backup took hours and hours. Almost a full day. But once that is done the incremental backups are fast. 15 minutes each night or less. And the Notification system in OS X lets me know the backup was successful.

For my system drive, I have a cheap USB bus powered portable drive. Something I found dirt cheap. Again, 1am every night, my system drive gets backed up.

I hear talk of off-site backups, and yeah back in my IT days we did that for large corporations, especially the off-shore manufacturers we serviced (southeast Louisiana is a lot of off-shore drilling rig business). But for a small shop, especially in your home, I’d not sweat an off-site backup option. It is appropriate for certain situations, but not all situations. Balance safety with being realistic. What is the likelihood that the building your working in will catch fire and burn down, or get caught in a flood and soak all your stuff for a week?

Fire-proof safes aren’t an answer to off-site backups, because they get hot enough inside to melt plastic (and hard drive platters/circuit boards), but paper won’t flare up. So screw fire-proof safes for storing backups.

Side story; we once had a client who’s offices caught fire. This was almost two decades ago, so tape backups were the best alternative for very large data amounts to backup quickly. They put them in a fire-proof safe. The backup tapes were melted. The computers themselves were brought to our offices. They were caked in wet soot. The interiors were the worst. Battery backups keep the computers running when the power went out, while fire fighters were hosing the building down. So the cooling fans suck all that wet soot into the interior, coating everything as if it had been thoroughly spray painted. The smell was horrid and powerful, thus we kept the computers outside.

We pulled out all the hard drives, and threw the computers into the dumpster immediately. Sprayed some sort of aerosol liquid cleaner on the to cut threw the soot, but not damage the controller boards. Back then, many drives still had controller boards on the outside of the housing. It stank and was a sticky, gunky mess!

Trivia time; cleaning mainly got rid of the smell and mess they were covered in. But more importantly, soot is a great insulator, and can cause a drive to overheat very quickly, no matter how many fans you have blowing over it.

After letting the drives air dry over-night, we stuck them in the new computers we build overnight for the client, and poof, they all booted right up with no issues! Of course we replaced all those drives with new ones. But we kept those drives in some test computers we had, and they actually lasted a good while. Couple of them died really quickly, in like days or weeks. But about half of them lasted 6 months to a year after that. SSD drives would NOT have survived the heat, the circuitry simply isn’t as stable as the electronic charges on a spinning platter. Everything in IT is a trade-off.


IF YOU NEED HELP!

So back up your stuff! It isn’t a matter of “if”, but a matter of “when”. PLEASE
contact me if you need any help with getting a reliable backup system in place. I’ve worked tons of systems, large and small, for over two decades plus, and am here to make sure everyone has a safe backup that is automatic.


REGULAR MAINTENANCE!

Performing regular maintenance on your system will go a long way to making your data and drives last longer. Plus, if there’s an issue with a drive going bad, it will rarely give you symptoms ahead of time. These maintenance routines can sometimes make those issues become more apparent sooner.
  • Daily - Run backups of all drives. Automated systems like Carbon Copy Cloner are best, because you don’t need to remember anything, it just happens, usually in the middle of the night so it won’t interfere with your work.
  • Weekly - Run Disk Utility to repair all drives.
  • Weekly - Verify all drives have at least 15% of their total capacity left as free space. Anything less than 10% can cause very odd behaviors that won’t fit any easy troubleshooting category, and you could risk losing data. I’ve seen this first hand on MANY occasions.
  • Monthly- Run Disk Warrior to repair the directions on all drives. This goes for spinning disks and SSD drives both. Corrupted directories are the single most common cause for drives to be labeled as damaged. If the directory goes bad, it’s rare to recover any data, at least not cheaply nor quickly.