Barwick video-trial

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.



Media madness I:PARADISEC video trial 2006

Media madness I:PARADISEC video trial 2006

Linda Barwick

Sydney Conservatorium of Music

University of Sydney

DELAMAN IV, London 2 November 2006

Linda Barwick

Sydney Conservatorium of Music

University of Sydney

DELAMAN IV, London 2 November 2006

2 November 20062 November 2006 Barwick - Media madnessBarwick - Media madness 22





Collaborative digital research resource set up by University of Sydney, University of Melbourne & Australian National University, 2003. (UNE joined 2004)

Collaborative digital research resource set up by University of Sydney, University of Melbourne & Australian National University, 2003. (UNE joined 2004)

75% funding from Australian Research Council LIEF Scheme (3 successful applications)

PARADISECPARADISEC






Why video?Why video?

From 1980s, portable battery-powered video cameras increasingly used by fieldworkers

Primary data for ethnographic studies of gesture, dance, performance, signed language, social interaction

Highly valued by cultural heritage communities

From 1980s, portable battery-powered video cameras increasingly used by fieldworkers

Primary data for ethnographic studies of gesture, dance, performance, signed language, social interaction

Highly valued by cultural heritage communities






Threats to videoThreats to video

Obsolescence of consumer video formats (VHS, super-8, Hi-8 etc)

Media deterioration much more advanced that audio media (estimate 15 years)

Kept in private research collections rather than deposited in archives (hence less than ideal storage conditions, lack of metadata)

Obsolescence of consumer video formats (VHS, super-8, Hi-8 etc)

Media deterioration much more advanced that audio media (estimate 15 years)

Kept in private research collections rather than deposited in archives (hence less than ideal storage conditions, lack of metadata)






Deterrents to digitisationDeterrents to digitisation Lack of internationally agreed archival formats and

standards for digital video, hence researcher fear of making wrong choice

Difficulties of storage and management once digitised - huge filesizes difficult to manage on desktop computer

Lack of user-friendly and robust tools to allow researchers to analyse video data and collaborate

Ethical and rights obligations - fears of compromising security and anonymity of consultants by outsourcing

Lack of internationally agreed archival formats and standards for digital video, hence researcher fear of making wrong choice

Difficulties of storage and management once digitised - huge filesizes difficult to manage on desktop computer

Lack of user-friendly and robust tools to allow researchers to analyse video data and collaborate

Ethical and rights obligations - fears of compromising security and anonymity of consultants by outsourcing






Preservation and Digital SustainabilityPreservation and Digital Sustainability

the only way to guarantee sustainability of digital audiovisual media is secure mass storage with well-structured metadata and an ongoing digital migration strategy, using authenticated streaming for access.

Kevin Bradley, APSR Sustainability Issues Discussion Paper, National Library of Australia, 2005.

International Association of Sound and Audiovisual Archives (IASA) Guidelines on the Production and Preservation of Digital Audio Objects (IASA-TC04). Aarhus, Denmark: International Association of Sound and Audiovisual Archives (IASA), 2004.

the only way to guarantee sustainability of digital audiovisual media is secure mass storage with well-structured metadata and an ongoing digital migration strategy, using authenticated streaming for access.

Kevin Bradley, APSR Sustainability Issues Discussion Paper, National Library of Australia, 2005.

International Association of Sound and Audiovisual Archives (IASA) Guidelines on the Production and Preservation of Digital Audio Objects (IASA-TC04). Aarhus, Denmark: International Association of Sound and Audiovisual Archives (IASA), 2004.






Purposes of trialPurposes of trial

Feasibility study for video ingestion

select appropriate standards

test impact on existing systems

design archival workflows

model costs for future funding applications

Preserve significant collections

Provide access copies for researchers and communities

Provide test data for EthnoER online annotation projects

Feasibility study for video ingestion

select appropriate standards

test impact on existing systems

design archival workflows

model costs for future funding applications

Preserve significant collections

Provide access copies for researchers and communities

Provide test data for EthnoER online annotation projects






Equipment and dataEquipment and data

Outsourced ingestion to professional video-maker (Paul Cockram) using Final Cut Pro 4 on Mac computer, ingesting video data from professional decks over firewire 400 cable.

Video formats tested: VHS, Hi-8, miniDV

Video files saved on 500GB LaCie firewire drive for transport

Outsourced ingestion to professional video-maker (Paul Cockram) using Final Cut Pro 4 on Mac computer, ingesting video data from professional decks over firewire 400 cable.

Video formats tested: VHS, Hi-8, miniDV

Video files saved on 500GB LaCie firewire drive for transport






Formats and standards adopted for trialFormats and standards adopted for trial

Master copy: .dv standard (maximum quality coming across firewire into Final Cut Pro) (saved inside Quicktime .mov wrapper) - approx 15GB per hour

720x576 pixels, 25 fps, PAL

Titles: basic metadata plus test signal at beginning of file, “end file” title at end of file.

No chapter markers unless depositor provides list of timecodes for this purpose

Master copy: .dv standard (maximum quality coming across firewire into Final Cut Pro) (saved inside Quicktime .mov wrapper) - approx 15GB per hour

720x576 pixels, 25 fps, PAL

Titles: basic metadata plus test signal at beginning of file, “end file” title at end of file.

No chapter markers unless depositor provides list of timecodes for this purpose






Access copiesAccess copies Also sent to archive because of time taken to

process (will review later) DVD copy (MPEG 2) - compress item to approx 4GB.

Save as muxed .mpg file or disk image .img Chapter markers inserted every 5 minutes

Web copy (.mov or .mp4) frame size 320 x 240 (480 x 360) 30 frames per second keyframe every 300 frames data rate 40 - 50 KB/sec (320 - 400 kbps) or 100 - 120

KB/sec (800 - 960 kbps) for larger frame sizes.

Also sent to archive because of time taken to process (will review later)

DVD copy (MPEG 2) - compress item to approx 4GB. Save as muxed .mpg file or disk image .img Chapter markers inserted every 5 minutes

Web copy (.mov or .mp4) frame size 320 x 240 (480 x 360) 30 frames per second keyframe every 300 frames data rate 40 - 50 KB/sec (320 - 400 kbps) or 100 - 120

KB/sec (800 - 960 kbps) for larger frame sizes.






Issues (1)Issues (1) Damage to videos (dropped frames) may cause stall in

ingestion, need to set FCP to tolerate Some VHS tapes (1980s) had too many tracking errors

to digitise Unstable picture (both hi8 and VHS) leads to higher

filesize in mpeg-2 (image stabiliser needed?) Communication issues with contractor led to wrong

filenames being used in titles, hence requiring reprocessing

Compression to DVD and web copies very slow (8 hours to process a 3-hour file) , ties up machine - better done in-house by batch processing

Damage to videos (dropped frames) may cause stall in ingestion, need to set FCP to tolerate

Some VHS tapes (1980s) had too many tracking errors to digitise

Unstable picture (both hi8 and VHS) leads to higher filesize in mpeg-2 (image stabiliser needed?)

Communication issues with contractor led to wrong filenames being used in titles, hence requiring reprocessing

Compression to DVD and web copies very slow (8 hours to process a 3-hour file) , ties up machine - better done in-house by batch processing






Issues (2)Issues (2) Data deluge at time of HD shuttle

500GB needing to be copied onto system - took a day swamped storage (ill-prepared) server partitions 200GB - not large enough, hence loss

of time in copying across partitions Backup scripts at Sydney and APAC became out of

sync, leading to data not being written to tape in Sydney, or being removed from server before APAC could mirror

timeconsuming diagnosis and repair, but systems now redesigned

Data deluge at time of HD shuttle 500GB needing to be copied onto system - took a day swamped storage (ill-prepared) server partitions 200GB - not large enough, hence loss

of time in copying across partitions Backup scripts at Sydney and APAC became out of

sync, leading to data not being written to tape in Sydney, or being removed from server before APAC could mirror

timeconsuming diagnosis and repair, but systems now redesigned






Transcoding for web deliveryTranscoding for web delivery

EthnoER project working with CSIRO’s Annodex platform to deliver streamed video

Uses Ogg Theora - best transcoded from full-resolution version (currently done by CSIRO)

Online delivery of annotations via EOPAS (EthnoER online annotation standard) - developed with University of Queensland Vannotea team building on Michel Jacobson’s ITE work (LACITO)

More details at our December conference!

EthnoER project working with CSIRO’s Annodex platform to deliver streamed video

Uses Ogg Theora - best transcoded from full-resolution version (currently done by CSIRO)

Online delivery of annotations via EOPAS (EthnoER online annotation standard) - developed with University of Queensland Vannotea team building on Michel Jacobson’s ITE work (LACITO)

More details at our December conference!






Costing and feasibilityCosting and feasibility

Final report on data trial in preparation - we will post on our website and blog

Costing only of production (staff time, consumables) - model at this stage does not take into account long-term storage and networking costs, equipment deterioriation

Funding application for equipment in 2007 unsuccessful - we will regroup

Final report on data trial in preparation - we will post on our website and blog

Costing only of production (staff time, consumables) - model at this stage does not take into account long-term storage and networking costs, equipment deterioriation

Funding application for equipment in 2007 unsuccessful - we will regroup






Video data in PARADISEC collection (30/10/06)Video data in PARADISEC collection (30/10/06)

0.5% of collection files account for 19% of the data volume

0.5% of collection files account for 19% of the data volume

Type Files Size (GB) Avge (GB).dv 23 23.44 1.019.img 8 24.750 3.094.mov 61 485.53 7.960.mp4 6 2.22 0.370.mpg 19 11.92 0.627TOTAL vid 117 547.86 4.683TOTAL collection 21,098 2909.24 0.138% 0.555 18.832






Further information:Further information:

Thanks to:Thanks to:Participating UniversitiesParticipating UniversitiesAustralian Research CouncilAustralian Research CouncilAustralian Partnership for Advanced ComputingAustralian Partnership for Advanced ComputingAustralian Partnership for Sustainable RepositoriesAustralian Partnership for Sustainable RepositoriesGrangeNetGrangeNet

http://paradisec.org.auhttp://paradisec.org.auhttp://blogs.usyd.edu.au/elachttp://blogs.usyd.edu.au/elachttp://ethnoer.unimelb.edu.auhttp://ethnoer.unimelb.edu.auhttp://conferences.arts.usyd.edu.au/index.php?http://conferences.arts.usyd.edu.au/index.php?cf=11cf=11

Documents

Barwick video-trial