Workflows - File Based Production
The considerations of file-based production | Training | Workflow | Camera Choice
Media | The Shoot | Post Production | Archive | Support
Introduction
If high definition has been the big story in broadcast technology in recent years then 2009 is set to be the year that file-based production took off. Increasing levels of HD commissions from broadcasters coupled with the emergence of a new breed of broadcast quality tapeless cameras has made file-based HD a reality.
It’s not hard to see why. Recording onto high capacity plug-in memory cards removes the need for tape, VTR machines, and the time consuming and expensive process of ingest for editing at a stroke. For those willing to embrace this new way of working it offers efficiency both in terms of time and budget whilst unlocking potentially lucrative future archives.
Tapeless has already earned itself a reputation amongst some forward thinking commercials and music video directors because of its practical advantages in a very client-oriented industry.
But instant access to digital files without the back up of a hard copy represents an unnerving culture shift for many producers. One of the biggest hurdles faced when embarking on a tapeless production is the handling of data during production, post, and managing of archive material.
Planning of each of these elements is key to ensure a production realises the benefits offered by this new technology.
The benefits can include
• Time and cost savings for production
• Fast turn around in post production
• Reductions in stock costs
• The formation of a flexible, high value archive of material
As with HD, working with file-based technology requires an understanding of the entire workflow process from acquisition through to final deliverable. This section of the Image Forum site is designed to give an overview of some of the considerations of working with file-based formats.
The considerations of file-based production
Training
Working in a file-based environment requires a shift in mindset from all those involved throughout the process.
One of the greatest fears for a production moving to tapeless acquisition is moving to a filebased
workflow and away from the tangible nature of tapes. Having a production’s rushes stored on a
drive / disk or card at the end of the day rather than a tape for many can seem a scary proposition.
If however this is handled correctly holding rushes in such a manner can have a number of benefits, these include the ability to log shots clearly, review shots easily and assist others further down the production process.
The careful management of this data at all times is critical and any staff involved in this must have a full understanding of the techniques required. Training of these staff is key to ensuring that the workflow benefits offered by file-based production are achieved and also to avoid the potentially disastrous situation of losing data.
Currently manufactures offer a number of different tapeless formats each of which offer different
advantages and limitations to the user. Clearly with a number of formats available it is important to have a working knowledge of each before undertaking a project. Training prior to embarking on a file-based project is a must. This may be in house or organised by a third party training company or supplier.
Workflow
Before shooting starts in earnest, workflow must be tested end-to-end to iron out potential snags in the shooting, post, delivery and archiving process.
As with all formats the workflow should be designed from the ground up with the final deliverables in mind at all times. A programme may be straightforward or may have a number of final variations with both HD / SD copies required, multiple international versions and copies for 3rd party distribution.
With these in mind a production can start to plan how each stage of the workflow will look:
• Choice of camera including format and setting
• Management of data on site
• Agreeing a universal logging and metadata framework
• Post production route Online / Offline and compatibility of hardware
• Deliverables
• Archive management
Camera Choice
Your choice of camera will be guided by a number of factors, these will include items such as: deliverable specification, format compatibility, type of production, and of course budget!
Always check with your commissioning broadcaster that they accept your chosen camera system for the individual project.
Below are examples of some of the more common cameras and some of their technical characteristics.
Full size camera options:
Handheld camera options:
Media
There are currently a number of different tapeless recording media available depending on the chosen camera system. Each type of media differs in physical appearance and the features that it offers the user.
Below you will find a summary of the most common media and their characteristics:
The shoot
The management and organisation of data during the shoot will impact the workflow throughout the
production process.
In addition to video and audio files other information or metadata is recorded. This metadata includes basic information such as video format, time code and date stamp. However, additional information assignable on the camera can also be recorded greatly assisting the logging and
categorising of clips.
Prior to the shoot there are a number of decisions that need to be made, these include agreement
between production and post to ensure the management and labeling of data is completed in a secure and efficient manner. The data entered during the shoot must be assigned to the material all the way through to archiving so consistency and accuracy is key.
Areas that need to be discussed and agreed include:
• How data will be managed on location
• Who manages location data
• Assigning metadata on location
Managing data on location
There are a number of different ways of managing data on location. Your choice of data management will be influenced by the type of production, shooting volumes, availability of skilled crew, and personal preferences.
These options include:
Large quantities of Cards/disks
Employ a “shoot now transfer later” policy removing the need to transfer footage from cards in a
location environment.
Taking a large number of cards on location can be an alternative, this cost can be reduced by hiring cards. This is the usual procedure for disk-based production.
Direct transfer from media cards/Camera drives to removable drives
This transfer method offers users the opportunity to transfer material to drives whilst reusing a
limited number of cards on location.
This provides the fastest method of transfer with data rates
up to 500Mb/s. Training is essential to ensure that all metadata and XML/MXF wrappers are
transferred with files. This will ensure that file integrity will be maintained.
Use a proprietary transfer software program or script to perform a verified transfer or a checksum
on source and destination files. Some manufacturers offer a free download of their data
management software suitable for Mac or PC use.
Always back up to at least two (preferably three) separate locations.
Transfer will be via card reader or card slot - express slot (SxS) and PCMCIA (P2) or Firewire or
eSATA leads for Camera drives.
Transfer to drives with viewing/logging
The viewing/transfer software from Panasonic and Sony offers a number of advantages when
used on location.
This software operates on both PC and MAC and offers some of the functionality of a NLE system.
• Viewing and logging of rushes
• Viewing and editing of shot markers (Good/Bad Shot)
• Editing User metadata (These fields are visible to FCP and AVID NLE for sorting/categorising
clips)
Panasonic and Sony viewing software is a free downloadable resource from the manufacturer’s
websites.
Field unit transfer hardware (P2)
A field transfer unit called the P2 gear (AG-HPG20 or AG-HPG10) is available for the transfer of
footage from P2 cards directly to drives.
The unit works in a similar manner to a traditional clamshell unit and removes the need for the use of a laptop on location:
• Battery operated
• Viewing, browsing and metadata viewing using LCD screen
• Precise audit of transfer status and integrity with checksum
• Playout to HDSDI or Component (record from HDSDI with AG-HPG20)
• Acts as USB or Firewire host to enable direct transfer to drives
• Transfer times typically 30% to 60% of running time
• AVC Intra record/viewer compatibility only on AG-HPG20
Using a Non Linear Editing (NLE) system on location
A NLE system works very well on location in a file-based environment due to the ease of material
ingest on a system.
A NLE system can be used to either complete a rough cut, further enhance
shot selection or expand metadata etc. There are a number of considerations when working in
this way:
• Never work with master rushes – these should be preserved - use subsidiary copies with the
NLE
• A MacBook Pro is capable of handling 2 streams of DVCPro HD If special effects are required
overlays and comps can be achieved on site
• Animation sequences can be tested on location
• Media can be batch renamed on ingest
• Additional meta data tagging can be completed
• Check your NLE system can handle the latest codecs (AVC Intra 100 and 50 require latest
codec/software variants).
The latest versions of FCP and AVID are able to work natively with AVC Intra and R3D, but older
versions require a conversion, below are the associated ingest time of each:
• DVCPro HD ingests at 120% to 200% of running time
• AVC Intra 100 ingests at 80% to 100% of running time
• AVC Intra 50 ingests at 40% to 80% of running time
• XDCAM HD 35 ingests at 40%-60% of running time
• Ingest can be of part of clip rather than whole clip
Who manages data on location?
The individual responsible for overseeing the management of data on location has a key role in ensuring the security of your material.
Although manufactures have endeavoured to produce systems to minimise accidental damage and loss of material, this is still possible if the individual operator is not properly trained.
Generally the process is relatively simple and with basic training it can be completed in a secure manner. The complexity of the task will differ depending on the method you choose to manage data on location.
Practising data transfer prior to the start of the shoot is always advised.
Assigning metadata on location
Metadata is one of the most important and powerful elements of file-based production. It enables content to be logged, searched and organised, at any stage of a production.
All card and disc based systems have custom media and file naming capability. This enables users to both add data to the card/disk and individual shots captured on the media. An example of its use would be for a producer to quickly see all shots of a particular presenter from a particular day’s shooting.
It can be used by an editor to quickly locate a shot needed to complete a project with high shooting ratios. It can also be used to search archive material to recall elements of past productions needed for future use.
Prior to commencement of filming a workable and practical naming protocol should be agreed to ensure rapid identification, management and filtering of rushes. Once agreed these protocols should be distributed to anyone involved in the handling of media.
Elements that can be included in naming protocols can include filming location, time and date, camera info, shot description, shot markers, operator names etc.
Both FCP and AVID are now able to interpret Camera metadata, and utilise it in the editing process.
Post production
Both Avid and FCP are compatible with the most common file-based formats and codecs. Below is an example of the post production workflow using different formats.
Red Camera and SI-2K can be handled in post in a number of ways. Both FCP and AVID can work
directly with the camera media, however because of the large file sizes an offline/online approach is
normally employed.
Usually lower resolution offline files are created from the master media, the offline edit takes place and then a conform undertaken during which the master full-res files are re-linked to the offline EDL.
There varied methods of working with these formats in post The frame and file sizes are very large
compared to Standard and even High Definition and powerful hardware is required for real-time viewing of the media. E Early adopters of the technology will have tried and tested bespoke methods to ensure a smooth ride.
There are several considerations:
• It is essential to have full liaison between Shoot and Post from the early stages of production
planning to ensure that all parties know what to expect in terms of shooting formats, codecs, LUTs,
Data Management and deliverables.
• It is essential that the file-naming protocols and metadata are preserved to enable re-linking at the
conform stage to the Master RAW source media.
• The lower resolution offline files must be of sufficient quality to judge sharpness of the shot.
• RED, SI-2K and ARRI D21 all shoot wide latitude RAW files: to enable offline shot selection and edit decisions to be made it is usual to apply an adjustment or Look-up Table (LUT) to the images to display them on an edit display or monitor. The LUTs can also be applied to monitors on location. The LUT used for each set-up or look on set can be added to metadata or the slate, enabling this to be applied to the offline media prior to the edit.
Archive
Shooting using file-based camera systems creates large volumes of digital files that must be securely stored as either a requirement of a commissioning broadcaster, or for future exploitation by the production company.
Although programme masters are still currently being played out to tape for delivery the majority of programme rushes always remain in a digital format.
The careful cataloguing of rushes during the shoot will by this point have created a valuable archive that can be reused for future exploitation.
It is important to discuss and agree an archive strategy before embracing file-based acquisition. File sizes within HD are very large and will fill a company’s storage network very quickly. A company shooting tapeless HD must plan a method of archiving this material in a way that remains efficient and cost effective:
• Plan for archiving and retrieval of rushes and finished programme material at the end of the
project
• Ensure archive method offers sufficient security for valuable rushes and finished projects
• Storage of media on SANs and local storage is expensive and detrimental to efficient operations
• Consider other options such as LTO-4 or other secure long-term archival methods – but ensure
media is adequately catalogued to ensure efficient retrieval
• Investigate cataloguing tools such as CATDV to ensure you can search and retrieve archive
material quickly and cheaply.
Support
New technologies are changing the way in which programmes are made.
As of yet there are no universal formats agreed between manufactures which causes challenges for production as they try to understand how each of these formats fit with their way of working.
Support can be found online from each of the major manufacturers:
www.sonybiz.net
www.panasonic-broadcast.com
www.avid.com/p2
www.apple.com/finalcutstudio
Image Forum