DATA WRANGLING ON LOCATION AT THE METROPOLIS STUDIOS, LONDON
Last week I did some data wrangling or copying and backing up camera rushes to external hard drives on location between takes. I was working in the Metropolis studios which is a famous music studio in London that has had Rihanna, Lady Gaga, Beyonce and Chris Cornell through it’s doors.
Metropolis Studios interior
The programme was Ontrack with SEAT and I was hired for the programme about ‘The Horrors’ and ‘Bombay Bicycle Club’, both popular bands. Here’s Ontrack with SEAT ‘Rizzle Kicks’ who are really good. The programmes i worked on haven’t been broadcast yet.
Ontrack with SEAT - Rizzle Kicks
Anyway I had a 4GB imac in the canteen with two G-Tech drives and I would wait around watching the performances, directed by James Russell, in the studio until a take was finished. A camera assistant would then hand me the four compact flash cards from the giant Sony F3 cameras and I would run upstairs and import them into the computer via compact flash card readers.
Each card took ten minutes to copy (I copy via the copy and paste method rather than the apple endorsed drag and drop as I was told it was more reliable. Now I don’t know if that’s true or not but why take the risk? (Although probably blindly following someone else’s advice is how bad habits are passed on.))
After each card is copied it was up to me to format them. Which is nerve wrecking! So like in most tense situations you need to get organised. I had the cards yet to be ingested on the left and the card were all really clearly labelled. ‘A2’ stood for camera A, second card for example and is one of the reason why camera assistants are invaluable, they organise everything to and from the camera! if I had got unlabelled flash cards and I accidentally knocked them over it would have been horrific.
Once I copied the media to a G-tech drive and once finished I needed to check that all the media was copied without corruptions. So I’d open up a finder window on the card in the card reader and compare it to the finder window of the drive with the freshly copied media. I’d count the files to see if they were the same amount and also to see if all the files appeared as picture thumbnails. If any of them resembled file icons with a ‘Q’ quicktime icon on them I knew that they were unreadable and therefore incomplete or corrupt and needed to be flagged up.
Lastly, I pressed ‘command + I’ to ‘Get Info’ on both the compact flash card and the copied folder to see that they were exactly the same size. Always copy everything from the cards as if you pick and choose you can disrupt the file structures which stop certain log and transfer software accessing them.
Once all the checks were done I dragged the files from the cards into the trash and ejected them. I then tore in half the white tape on the cards that said the camera and card number, which told the camera assistant that I had already ingested them, and gave them back. This was repeated for about twenty cards and had to be done in a fast turn around as the cards were often needed for more filming. I’d often be on the last card of the batch and someone would be there to collect them.
Data wrangling is a straight forward job but the stress is a lot because you know that if you miss out any of the cards then it’s on your head. While everyone else goes home you are the one that is waited from the phone call from the editor… “Oh hi mate… I’m looking for Camera D card 3 and I can’t see it…” Enough to make you physically sick.
On big commercial shoots DIT’s use more thorough methods such as ‘Checksum’ programmes that copy media and then run analytical programmes on what’s copied. Checksums inspect every byte copied but take hours and hours to run their course. They are fool proof but for many jobs the editor needs to start working the next day, so they are just not practical.
Anyway once a batch of cards went back to the cameras, I needed to convert them all to Apple ProRes (HQ) for editing processes. Now this process takes AGES so I had a number of programmes to use’Mpeg Stream Clip’ (takes 1.25 times footage duration to convert), Compressor ((takes 1.5 times footage duration to convert) or Final Cut Pro Log and transfer (takes 2 times footage duration to convert as it’s not a dedicated encoding programme. Takes too long). I go with MPEG Streamclip (a free bit of software from the web) as it’s the fastest, you can also batch encode and you can also choose how many to encode at once out of a maximum of four files at a time. The problem with it is that it strips the timecode which is a problem if the editor is going to be synching their clips vis timecode for multiclips. James Russell’s crew were using clapperboards for synching so I was fine. I I opened MPEG Streamclip, went to list in the toolbar, and clicked on batch list. This allowed me to highlight all the clips and export them as Apple ProRes (HQ). Then I just chose were I would be saving the ProRes files to and then all the files appeared in a list. All I then had to press was ‘Go’ and sit back for four hours and then everything is finished. It’s quite galling to know that while the crew leaves at 5pm you will still be there at 10pm.
Anyway it was exciting to see James Russell and the camera crews work and I hope to be hired again.
I hope this post will be of some interest for any new data wranglers out there…