Evolution of the technical and professional skill sets and roles within the digital audiovisual archive sector, Jean Varra, INA

FIAT /IFTA Conference Amsterdam 2014

Evolution of the technical and professional skill sets & roles within the

digital audiovisual archive sector

Introduction: archiving scene and jobs in transfer

01

technical skills and roles | 6 octobre 2014 3

Introduction :archiving scene and jobs in transfer…

Digital technology entered the television domain about 20 years ago replacing progressively and unavoidably analogue television.

Digital has introduced lot of changes in the audiovisual domain and therefore archiving: new technical systems, new uses, new delivery modes….

Today:

•Production and post production systems have totally shifted from tape based workflows to tapeless workflows

•Files have replaced analogue and digital video tape formats

•Servers and networks have replaced video tape recorders

•A high percentage of content produced today is digital and volumes grow faster and faster

•In a IT world, automated systems play an important role data management and data control have become key factors.

Introduction :archiving scene and jobs in transfer…

Though, fundamental functions of Archives remain the same: acquisition, preservation, access, documentation…roles and skills change.

Some traditional skills and roles are disappearing, some have evolved, new ones have appeared or are still appearing.

The dilemma is that archivists ,while trying to follow and assimilate/digest the faster and faster changes in digital technology, must still continue to keep alive some traditional skills and roles.


Introduction:archiving scene and jobs in transfer…

The following presentation will try to answer the following questions:

• What are the major evolutions of the technical side of archiving : what has already changed ? What is changing?

• How current skills and jobs are(will be) affected by those changes ? Which main topics archivists must take care of ?

• What are the required new roles and skills?


Archiving functions in a digital world

02

Ingest

Store document

deliver

Archiving functions in a digital world

In order to define the technical skills and roles in a digital archiving system, it is interesting to use a diagram based on the OAIS - Open Archive Information System model which details the main functions and, map on it the required skills and roles.


8

Ingest

Access /deliveryArchival

storage

Data manageme

ntProducers/ sources

Consumers / users

QueriesQuery responsesorders

Descriptive

information

Descriptive

information

Master/viewing

file formats

Master/viewing

file formats

Administration

Preservation planning/management

film

V/A tapes

files

Delivery

formats and

media/ browsin

g formats

technical skills and roles | 6 octobre 2014

MP3

Ingest entity

03

MPEG1

MPEG2

H264

JPEG

HEVC

DV

DVC pro

MXFMOV

AVI

AVC intra

WAV

BWF

AAC

AC3

DIVX

2 inch

1 inch B

1 inch C

film

Betacam

SP/SX

IMX

JPEG 2000 TIF

From analogue to digital…

Analogue to digital processes can be considered as part of the ingest entity.

There are still many film and analogue video formats over the world to preserve and digitize but also early digital video formats now obsolete.

To ensure a successful film digitization 2 types of work are required:> Film restoration or mechanical repairmanual work>Film transfer and color correction Evolving from telecine to scanning operation

To ensure a successful video (audio) tape digitization are required:>A good preparation of the tape is the first important criteria to preserve the quality

of a content: cleaning, baking and repair…if needed>a good playback of the original tape is the second criteria to preserve the quality of

a content. requires tape players in good condition


From analogue to digital: roles at risk

Concerning these activities ,there are several traditional roles at risk:

Film restoration: > Less and less people are interested in this job> Film laboratories are closing and disappearing less and less opportunities to work on film media.> Only large digitization of film could make this job last

>More generally skills and knowledge on film media are disappearing.

Video playback:> Main problem to face is to maintain old obsolete machines.> maintenance skills and knowledge on old tape machines 2 inch 1 inch are rapidly disappearing

combined with lack of spare parts.> Almost impossible to motivate young people on these old technologies> To play more recent tape formats (Beta, Beta SP,DVCpro… ) there are still possibilities to find

machines and spare parts. how long ?

>More generally, video tape recording (even digital) and associated role (video tape recorder operators and maintenance technicians) are inevitably disappearing in the broadcast and post production field and therefore in archives.


From video to file…

Encoding is the function allowing conversion from either analogue or digital video signals to file.

A file is a combination set of :>Essence ( image, sound) ie MPEG2, DVCpro, JPEG2000, WAV…>Metadata: technical metadata, identification metadata… >Wrapper:/container MXF, MOV, BWF,AVI…

To preserve quality of the original content, the choice of the destination file format is essential.

A file is defined and set by parameters, and technical metadata which are essential to make it usable and understandable.>spatial resolution ,bit rate, GOP, field order, interlaced/ progressive scanning, aspect

ratio, audio tracks….

A file must be at least standardized but this is not enough to ensure interoperability.


File format conversion/transcoding

File format conversion/transcoding system is a central function of the ingest entity but also for the access entity.> In the ingest entity, the transcoding system is used to convert various file formats

(ephemeral, experimental, old, proprietary…) into others file formats (for example into a master/mezzanine file format or a viewing format)

>In the access entity, it is used to generate file formats compatible with diverse receiving platforms for diverse uses.

File format conversion is required to solve interoperability problems.

But, transcoding is neither a “transparent” or a “so easy” task:> It may introduce some quality problems or impairments on the content.>Even if a file is “well defined” , some interoperability problems may occur:

> A file may be corrupted> Some parameters of the file may not be taken into account by a specific device or tool> A file may be accepted/perfectly used by one tool (transcoder, editing system, players…and not by

another according trade marks and standard implementation…)> Unexpected problems

How to face these problems?


A new role: « file specialist »

Encoding and transcoding in the digital process require very skilled and trained people , “file specialists” :


Main roles To determine and set the best parameters to ensure quality and sustainability of the file. to test/check interoperability with tools of the market.to analyze, understand and solve all potential interoperability problems and set corrections if requiredSet the parameters of quality control systems

Main skillsHe/she must perfectly understand the deep structure of the various files and the role of each parameter the characteristics of various coders/transcoders, file players … He/she must be able to understand, analyze specific problems and bring a solution. ( tools are required) .

A validation function: quality control (QC)

In the past, QC consisted in checking video and audio on video tapes.

Because processes are more and more automated, most important work for AV technicians has become “monitoring systems and performing QC”

Before storing and archiving files, quality control is required in order to make sure that :

> During digitization, the play back of the original media has been performed successfully ( checking levels, clogging, drop outs sound presence….)

> The file structure, essence or wrapper is OK / not corrupted> The parameters of the generated file are respected.> The files are correctly named ( identification)> The coding and/or the transcoding systems have not introduced new impairments.> The content is complete (beginning and end) but also that received files correspond to the expected ones.

There are now automated tools able to check content quality and syntax of files delivering time coded reports that can help an operator and make him focusing on the detected problems

> Baton (Interra systems), Cerify (Tektronix)…

If a QC operator cannot analyze a problem, he/she can ask for help from file specialists


QC operator


Main rolesDetect what are the artifacts due to the original and/or its playback ( general or specific)

Detect what are the artifacts specific to coding or transcoding

Detect syntax/structure problems

Analyze reports from automated QC

Propose corrective actions

Main skillsAbility to analyze image and sound Know and understand what are the artifacts due to the original and/or its playback ( general or specific)

Understand file syntax problems know what are the artifacts specific to coding or transcoding

Understand reports from automated tools

Archival storage entity

04LTO digital library-Ina

Archival storage entity

Because of the ever faster growth of data volumes and diversity of uses , a hierarchy of storage ( off line, on line, near on line) is generally implemented.

A hierarchy of storage requires some management rules to set and follow.

Archival storage management requires a broad IT knowledge in servers and networks :hardware/software and data administration, hard drive, tape robotics, and corresponding software (HSM….) The required skills of an archival storage manager are more or less those of an IT administrator

In large companies, tasks may be split between several IT specialized roles and responsibilities: hardware engineer, network engineer, security manager, help desk…


Role : storage manager/engineer


Main rolesCommon/usual tasks performed by a digital storage manager are to:

Receive, provide/deliver , to delete, replace, back up data organize data, collect technical metadata (produced at each step of the process)Warranty operation :acquisition and delivery…

He/she also monitors permanently storage activity status ( streams, capacity, and performance… )

Objectives are to prevent system from failure and data losses use of software to monitor data tape libraries for example RVA ( read Verification appliance) in data tape libraries.

Other important tasks are :To check/ensure data integrity ( prevent data from corruption, calculate check sums…) and security ( backup data and setting disaster recovery function and processes)To plan/set strategies ( according to quantities) and perform data migration from one type/generation of storage to another ( for example from LT04 to LTO6) Assume scalability of the systemPrevent system from obsolescence: technological survey and planning evolution

Main skills

Understanding networkingUnderstanding operating systems and related files systems.Understanding storage architectures (NAS, SAN, HSM….) Undersanding file formatsSoftware /hardware installation Development/Programming

Access entity

05

H264

JPEG

HEVC

MOV

AVC intra

WAV

BWFMP3

AC3

JPEG 2000

Pro Res

DnX

JPEG

MXF

WAV BWF

TIF

Access entity

The objectives of the access/delivery entity is to provide technical services as “simple data formatting” :

>file format conversion to deliver the expected file format.>Partial restore / extract creation: to deliver a sequence,> Logo or burnt in time code insertion

But also “advanced” formatting functions like: (when required)> publishing, editing functions, digital restoration…

Access activity: queries request and queries response, orders

Delivery on various modes: >copy on digital video tape (less and less)>Files on IT media: removable hard drives, data tapes, memory sticks and cards…>File over networks (more and more)

Ensure quality of what is delivered

Protect data.


Acess entity roles

Simple delivery can be implemented in an automated process/workflow.

(Request through tools allowing viewing/ selection/setting time code in and out/order, buy, deliver)

:


Main rolesMonitoring the activity:

check if customers orders have been satisfied on time (for example, if it is linked to an urgency news)

Monitor if the technical system is OK. If not, alert concerned department( for

example IT department if a server a network has failed e or the file specialist)Monitoring the delivered quality and apply corrections if needed

of what is delivered (has been delivered) with the help of automated tools , Analyse the problem and correct if possible or ask to a file specialistSecure the delivery services

In case of urgency or if the technical problem lasts, use alternative delivery modes to deliver the expected content.

Main skills

Understanding installed processes Understanding files and their problems and possible solutions see ingest entity skillsUnderstanding delivery modes

Access/delivery entity roles

Sometimes, the delivery of a content requires some specific formatting actions ( ie. which cannot be automated)Editing, post production ( subtitling, dubbing…) « Advanced » Up conversion--< reframing the picturesColor gradingAuthoring for DVD, Blu ray, publishing…Digital restoration of image or sound

… All these jobs require specific «technico-artistic skills ».

Digital restoration is an increasing need in order to match today quality needs and give new opportunities to archive contents.

In order to respect the original work, digital restoration requires not only to master the tools but also a to have real knowledge of history of production, the way it has been produced …


Other access skills and roles

Data protection management according to the adopted protection system ( signature, water marking…)

>Marking content>Monitoring use of content>Detect illegal use

Publishing over multi platforms: Web masters…>Web>VOD>Mobile>I tunes>…

Data mining


Temporary conclusion

06

Summary: roles


Technical roles

Traditional roles: handling old media

Still required in archives but regularly decreasing

File specialist New essential role in the jungle of files

Monitoring activity and QC control

main new roles for AV technicians

Data /storage management Central technical role of a digital archive

Linked roles Publishing, data protection, data mining, digital restoration, post production…..


Ingest includes knowledge of:

Old media use and behaviour

file formats and their transformation

record keeping systemsprograming languages

database administrationvalidation tools including

QCIdentification

Preservation needs knowledge of :

Technological history of the archiveoriginal record/ media either analogue or digitalOriginal tools (ie tape player or original operating system)Migration between operating systems ( for born digital records)

Storage requires to understand:

Operating systems and related files systems

Management actions like Software installation

issuesSecurityBack up

Data mining

Access needs to understand:

Networks and distribution infrastructures and modes (internet, VOD…file formats and their transformationValidation tools (QC)Protection tools

Management requires:

To understand broad technical and IT concepts and their

applicationTo plan and evaluate solutions .

Bring together a team of experts of various disciplinesMake people speak the same

language Digital archivists?

Summary: knowledge

Some axes of technical training…General Knowledge managers + all

Practice of file formats and AV compression:> Fundamental knowledge of standards , formats and uses. Quality parameters and

comparisons

Understanding video ad audio for computer people

More specialized trainings: file specialistsOptimization of compression:

>Setting parameters of different formats and tools ( encoders/transcoders) to get the best results, automation of tools, analysis tools…

Specialized trainings on each format ( MPEG2, MPEG4JPEG2000, HEVC, MXF, DCP..>Deep knowledge of parameters and related problems, performance analysis.

Quality control:Basics of quality control of picture and soundPractice of quality control tools: of Baton, Cerify….

Media server administration: storage managersServer technologies and architectures, exchanges , security of

data, users profiles….technical skills and roles | 6 octobre 2014 28

Thank you for your attention

[email protected]


Presentations & Public Speaking

Evolution of the technical and professional skill sets and roles within the digital audiovisual archive sector, Jean Varra, INA