Video Production Storage Explained

There are a lot of questions folks ask about “storage” in relation to video editing. It can be confusing! USB 2, USB 3, Firewire, Thunderbolt 2, Thunderbolt 3, HDD, SDD, bandwidth, capacity, etc. In this blog post I hope to simplify this, and give some maintenance tips.


line


Video Production Storage Explained

Basically we’re talking about drives, where your media and Libraries are stored for FCPX. This is the slowest part of your computer system, this is where most data pipe bottlenecks happen. So let me make an attempt to explain this in an easy to understand manner. I will not get overly technical, as that will not really help in this context. Bear in mind I am a retired IT engineer and am familiar with the technology. I am also a trainer and consultant with over a decade behind me and have developed what I consider a good sense of what will and won’t work in real-world situations. But I’m not omnipotent and all knowing, I don’t want to be omnipotent and all knowing, there are things I simply do NOT want to know about, so don’t ask me the meaning of life.


1- Drive Types
There are basically two drive types. The first is the hard disk drive (HDD) that have lots of moving parts. They have platters (where the data is actually stored) spinning at mind boggling speeds, and read/write heads that literally float on a sea of air over those platters, and are carried along on an arm that swings back and forth. Picture an old vinyl record player, but the needle never actually touches the record, it floats on the air current produced by the record spinning very fast. For these, ONLY use 7200rpm drives, as other slower speeds won’t read/write data fast enough for video editing.

The second type of drive is the solid state drive (SSD) that has no moving parts, outside of the electrons racing through it at super-mega mind boggling speeds. These are faster than the HHD drives mentioned above, but are also much more expensive. Back in the day when they first were introduced to the computer industry, it was preached that these SSDs would come down in price to match the price of HDDs when they became more widely used and more were made. Well, that day has come and gone and they’re still very expensive. Plus you can’t get them as large as you can HDDs.

So SDD is faster and more expensive than HDD, but which you need depends on your situation. As an internal system drive, SSD is the best option always. As an external secondary drive, it all depends on which fits your budget and needs. More on this later.


2- Configuration Types
There are two configuration types. The first is a stand along drive, which is only as fast as the drive itself can receive and send data, and the speed of the connection to the computer.

The second is a RAID array. This is when drives (in even number groups) of 2, 4, 8 or more drives are configured together to do two things. First it makes them faster. Data is written and read, spread across the drives evenly. Each drive can write/read at its own top speed. All together they’re writing and reading more data faster as a RAID group. This speed is also regulated by the type of connection to the computer.

RAID is faster and more expensive than a stand alone drive, but which you need depends on your situation. For portability, you’d want a fast stand alone drive. For your studio desk where you process lots of jobs daily, a RAID could be the best option.

3- Connection Types
Today there are only two types of external connections, and each of those come in generally two speeds. I am referring to what is popularly available from resellers and is realistic for purchase, not what actually exists. The two types are USB and Thunderbolt.

USB comes in two popular speeds now; 2.0 and 3.0 which are version numbers of the technology. Obviously 2 is slower than 3. USB 2 will handle HD video pretty well, and struggle with HD multicam work (if the connected drive itself is fast enough). USB 3 is much faster and will handle up to 4K video, and struggle with heavy multicam work.

Thunderbolt comes in two popular speeds now; 2 and 3 (just like USB) which are version numbers of the technology (just like USB). Both are ultra fast and will handle editing up to 4K and 5K easily (if the connected drive itself is fast enough). 8K could still struggle depending on the drive you have connected. I’d recommend T’bolt 3 connection to a RAID 5 for 5K & 8K video work.


4- Stand Alone versus RAID
A stand alone drive is nice for portability. Small, fits in your laptop bag easily, can be fast enough for small video work. I use a 500 GB SSD drive, using T’bolt 1 (not even 2, it’s that old) for teaching FCPX and Motion, and for editing on the road in the rare times I need to. I also have a 1TB HDD, using USB 3 for the same purpose when I need to travel with more data. Both work fine for those situations.

Now, stand-alone HDD or SSD will only be as fast as the connection you use. T’bolt is faster than any stand alone drive, so it could be you’ll be wasting money on the extra T’bolt speed (bandwidth) when a USB 3 unit would do you better.

RAID is not going to be very portable because it uses 2 or more drives in one enclosure. That enclosure has a RAID Controller Board on it that manages what data goes where, what to do when one drive fails, etc. Each individual drive also has its own Drive Controller Board on it, which manages input and output of data and monitors/controls the functioning of all mechanical and non-mechanical parts. More about that later. A RAID is fast and comes in versions: RAID 0, RAID 1, RAID 5, RAID 6, etc. Trust me and stay away from anything labeled as RAID 1+0 as that is super outdated and modern RAID enclosures more than make up for the issue that dinosaur was created to deal with.

RAID 0 has no redundancy, RAID 5 and 6 do. These three are the most common versions used.
- RAID 0 is used mostly for only 2 drives and gives the best data speed theoretically. It has no redundancy, so if one drive dies, you’ve lost all your data (backups are not a luxury, but a necessity for all storage types).
- RAID 5 is commonly used for an array of 4 or more drives. This has redundancy so that if one drive dies, the data is backed up across the other drives. If more than 1 drive dies, you’re hosed, you lost all your data, so back it up, man! When one drive dies, the system slows down until you replace that bad drive and the new drive is built and ready to rock, which usually takes 12-24 hours. When using RAID 5, add up the capacity of all drives, minus one of them. Example; 4 drives, 4 TB each = 16 TB, minus one drive (16-4) is only 12 TB of usable storage space. That one drive capacity is taken up by the redundancy across all the drives.
- RAID 6 I’d not recommend unless you’re using 8 drives or more. It has redundancy and can lose 2 drives before all data is lost. When using RAID 6, add up the capacity of all drives, minus two of them. Example; 8 drives, 4 TB each = 32 TB, minus two drives (32-8) is only 24 TB of usable storage space. That two drive capacity is taken up by the redundancy across all the drives.
- SSD RAID enclosures are not as common due to the price and limited compact of SSD drives. But using “enterprise level” 7200rpm HDD drives make for great RAID systems.
- RAID drives like all others are also dependent on the connection type. USB 3 or T’bolt 2/3 are about all you’d use. If you’re using 4 or more drives, I’d recommend a T’bot connected enclosure.


5- Never Store Work On The System Drive
There’s a very good reason this is the common wisdom. A single hard drive, such as your system drive, has a limited bandwidth. Basically, how many bits per second of data can it write and read. The operating system (macOS) and your applications (FCPX, Motion, Compressor, LPX, Pixelmator Pro, etc, etc, etc.) are constantly writing and reading data from invisible “working files” on your system drive. They need enough bandwidth to do this quickly and efficiently so that your Mac (or PC) runs smoothly. If you pile the enormous bandwidth demand of video editing on top of that, the system drive’s bandwidth could clog up and slow things down.

Thus, all media and Libraries and project files and cache should go on a secondary drive. Hopefully the fastest one you can reasonable afford, that is matched for your work. This way the system drive has plenty bandwidth for the OS and apps, and this secondary “media drive” can use all of its bandwidth for your video editing work.

Video editing work is very demanding because each video clip, each audio clip, is a “live data stream”. That means is must read/write all that data in real time, which is a huge demand on a computer and its drives. If you’re editing a 6 angle multicam, there’s six video streams fighting for bandwidth. Imagine having six Vimeo videos playing live without hiccups over your Internet connection. Yeah, that’s gonna eat it up. Then your kids start to watch an HD or 4K moving on your Apple TV, and suddenly your Vimeo feeds get all messed up and the house blows up in a fireball of frustration, confusion, and defeat… Or maybe you just get a tad ticked off?


6- Drive Maintenance

6a- Drives, even SSD drives, die. That is the hard, cold fact of life. They die for a variety of reasons: the read/write head crashes physically into the spinning platters and damages both, other physical damage from a hard enough drop onto the sidewalk can cause it to malfunction, the drive’s own controller board can go bad (replacing it is expensive and doesn’t work) from defect/age/power surge, the drive’s own connection ports go bad (I’ve had that happen a lot with Firewire connections, good riddance).

Yet the single most common reason drives die is that the Directory gets so corrupt it flat out fails. Nothing physically wrong with the drive, but it’s realistically dead. Once a drive goes bad, you have no way to know why (scraping noise means the platters and head are rubbing). If you sent it in to a “clean room” facility to check it out, they’d charge you a kazillion dollars (USD) and the reason wouldn’t help you out one bit.

The Directory lives on the very outer rings of an HDD’s platters, and also inside of an SSD in a designated area of the chips. It is like a phone book, it tells the controller board where everything is. Need a file called “My Great Video.mov”? The computer asks the drive’s controller board for it, the controller board looks it up in the phone book (the Directory), then go to the physical locations on the platters of an HHD, or on the chips of an SSD, to gather it all up and deliver it back to the computer. No, a single file is not in a single location, it gets spread out over several sectors, tracks, blocks, which I won’t go in to here.

So if the Directory gets too corrupted, nothing can be found, it’s just a drive full of 1’s and 0’s that nothing can make sense of.

6b- The other issue with drives, most importantly system drives, is how much free space is on it. Remember I mentioned all those invisible working files? There needs to be enough free space for them to operate properly. If you have less than 10-15% of your drives total capacity as free space, you may encounter: system slow down, temporary freezes, total data loss for some files, and other symptoms that any computer tech will not know about, seriously, it can in some situations get weird, seriously weird.


6c- Thus here is my guide to regular drive maintenance to keep your data safe.

Daily
Have a nightly backup. Every night (or day) have a total backup done of your system drive first. That must be a bootable copy, too. Then back up each and every drive you work with. It is not about “if” but “when” your drive dies without warning.

Weekly
- Verify free space on all drives, especially your system drive.
- Run Disk Utility’s “Frist Aid” function on all drives.

Monthly
- Run Disk Warrior and repair the directory on all of your drives. It will require you boot from the USB stick they send you, and keep it updated. They do a good job of teaching you all of that. It’s really simple.


In Conclusion
If you have any questions, please feel free to post on my Facebook account, or email me directly. I hope this helps someone out there, and clears up some myths. I know this is far from comprehensive, but I just wanted to give the basics so you’d be armed with good knowledge to make good decisions, AND know when to ask questions, and who to ask. Don’t trust just anyone with technical questions. Cause someone has used a computer for 10 years doesn’t mean they know how it works inside. Now, go make your drives and data healthy, and Rock Those Edits!