Tested’s Media Management Workflow!

16 thoughts on “Tested’s Media Management Workflow!”

1. I have always wondered how Tested keeps track/archives so much content.
   Thanks Joey. Well done.

   Reply

2. So, you do a “push/pull” workflow, not editing on the NAS. At my current day job (working as a storage/Linux geek with CNN/Turner editors/producers), we use high performance storage to allow folks to edit on shared NAS. It lets us insure the snapshots and backups are done using the active projects and we’re able to do recoveries if/when something happens to the project by reverting to a previous version. It also makes handing projects off between different editors much easier (with ~20-30 editors, a few senior editors, graphics folks, etc. this is a big concern).

   Our footage ingest is done using workstations that have the storage mounted. It’s a manual process where the producer/editors drag and drop from the cards/direct attached disks into specific directories. They know the guidelines for the directories where they drop things and how they’re organized so nobody goes rogue because doing so makes everyone’s lives harder. It’s more labor intensive than the workflow you demonstrated, but it’s necessary due to the larger number of editors we have.

   We’ve gone from using old hard drives/tape to shoving that stuff up into S3. It does have per-month costs for maintaining storage (Glacier helps mitigate this a bit) and costs money to retrieve content back (Glacier exacerbates this a bit when you go over limits), but tends to be better than having hard drives/tapes in cabinets (which also means they’re vulnerable to physical damage/loss due to environmental factors).

   I’ll actually be moving out to the Bay Area in August for a new job. If you guys are up for media workflow discussions, I’d love to chat and find out some more of the history behind how you got where you are and see if I’ve got any good suggestions to help with expansion plans.

   And thanks for the look behind the curtain. I love these BTS segments!

   Reply

3. PERFECT time for me to see this feature. My world is more ProTools and Logic for audio productions but I am moving my way into animation projects. Thanks TESTED.

   Reply

4. My work flow for videos is still messy, but I’m getting better.

   Right now I just have a local copy on my secondary SSD, and then everything backed up on my OneDrive. I recently picked up a 1TB WD HDD to act as an additional local back up, but I haven’t gotten around to hooking it up yet. At the frequency I make videos that amount sound be plenty for quite a while. I think my next storage upgrade is going to be bigger SD cards. Right now I’m working with just a 16GB card in my video camera and either a 16 or 32GB in my DSLR.

   Giant Bomb’s new video gal Abby has some sort of video production/film degree, and she mentioned how it was drilled into her to give insanely detailed names for all of her video files. Up until now I’ve just kept the numbers my cameras give the files. But given how much dang B-roll I shoot, some of which looks exactly the same from the preview still, I think I should probably definitely start doing the same thing. (I don’t think I would do it retroactively however, as that would not only be a pain, but would also break my Premiere projects.)

   I also have a problem of not deleting files I don’t use. I mean, I’ll delete the really bad stuff. But, did I use a panning left shot? Well, I might as well keep the pan right one. What if I need/want them in the future?!

   Reply

5. Re the RAID setup, I would touch on a couple more things. When populating the enclosure, ensure the HDDs are not purchased from one source (get non-sequential serial numbers from separate sources, 50/50 is fine). Bad HDD batches do exist (manufacturing issues, shipping damage), leading to my second point. I would dedicate an entire drive as a hot spare, and also keep a dedicated spare drive on hand as a cold spare. Yes this means “losing” two drives worth of storage (1 to parity, 1 to hot spare), but this will safeguard against a cascade failure leading to complete data loss. Without a hot spare, the parity drive will allow recovery from a single drive failure, but what happens when a second drive fails before someone comes onsite to replace it, before parity is restored? *poof* All hard drives fail, the question is when and how close together. -Someone who runs various enterprise NAS, SAN and DAS systems.

   Reply

6. Joey,

   As someone who works with archived data, I enjoyed your talk on archiving video from Tested. Hey, we are all nerds here; I enjoyed any discussion on archives. Happy to see that you are doing this.

   As someone who works with archived data created by others, I hope that you consider those persons who will want to check out your work in 20-30 years – this is my life right now, check out data produced 20-30 years ago (and more). I suspect that the data producers had no ideas that I would want to check out their work.

   Will folks in the future have the ability to watch your videos given the technology 20-30 years from now? A climate scientist may want to check out your wonderful videos of Greenland. Does your archival system ensure that they can do this?

   Perhaps you might want to think of your work as a legacy describing the world now. No pressure 🙂

   Reply

7. Very useful, great idea. Will be putting this to good use!

   Reply

8. Thanks Joey, always interesting to see how other people deal with their data bulk – I recognize a lot of similarities in the way I work with the bulk of media files. Did I get it right and do you actually delete all of the footage you didn’t use in a project (for clients)?

   Reply

9. Sorry to be the first one to say this, but this storage solution made me cringe. Hard drives should be kept online at all time, not stored in a container. Having them online will keep them active and S.M.A.R.T checks will keep track on them in case of failure.

   So add the disks in a large server and let freenas and ZFS do the rest. And all the video will be online at all time. And get 10GB network cards and a few SSD and you are good to go.

   http://www.45drives.com/

   Sorry Tested, but this is really amateur.

   Reply

10. I couldn’t agree more! I manage the special effects archives for Steve Johnson. We back up everything to a Synology RAID. We even went so far as to have two hot spares in the RAID as well as several cold spares. Very early on, we had three drives fail at once (same source, sequential serials). Months of work scanning and digitizing old VHS tapes would have been lost had we not had yet another backup. We narrowly dodged the bullet but learned a very valuable lesson.

    Synology devices are amazing and very affordable. They have active monitoring and alerting of the drive health (something we were not aware of initially but now use). I now have three of these, backing up to each other in different locations as well as slow backups to Amazon.

    Reply

11. I would be super interested in seeing the directory structure and file organization you guys use. It’s a constant struggle, and always has been, from my days at CafeFX to my humble little animation studio, data is always a challenge.

    Reply

12. Just wanted to thank you for a great piece. It had just the right mix of overview and detail. I’m the senior producer for the University of Windsor in Canada. Like everyone, I’m drowning in data. I’m going to use so much of this as we reorganize our media systems. Please do more.

    pete

    Reply

13. I’ve been wanting to get a Synology for a safer backup. All of my hard drives are aging, so it’s just a matter of time before they fail. I was thinking of setting it up for a 2 drive failure (just in case), or is that overkill?

    Reply

14. Hi Joey, just wanted to add that in the video, you said you turn the DS1817+ off to replace the drive…

    You don’t need to.

    The bays are hot swap. All Synology that have the drive release on the front are hot swap, its only the cheaper ones where you need unscrew the case that need to be powered down.

    Also not mentioned is the naming scheme, the 1817+ is a “Plus” so faster CPU and 3 year warranty, its the 2017 model and finally, the 18 means it can store 18 drives. You can buy a expansion boxes to increase storage, this model can add 2 x DX517 to give more drives.

    As others have said above, I use 2 disk fault tolerance, instead of one, its a small price to pay for extra security. I also use the SSD cache to speed up reading and writing to it.

    Finally, in the next few days DSM6.1.3 should come out that finally fixes all the Spotlight indexing issues.

    Reply

15. Joey’s videos are pretty much one of my top favorite segments on the site.

    Reply

16. Joey, Any chance you can do a run through on your travel protocol for equipment and how you choose or source your equipment when you guys go out?

    Reply