Managing the IntangibleBy Steven PugliaPreservation and Imaging SpecialistU.S. National Archives and Records Administration8601 Adelphi Road, Room B810College Park, MD 20740, USAPhone: 301-837-3616Email: steven.puglia@nara.govApril 2007

We no longer have a choice - compared to 4years ago:

•Conventional methods of analogreformatting are becoming unavailable orobsolete for many types of records and formats.

•User expectations have changed as well -people expect all historic resources to be online.

•We are and will have to use digitization forpreservation reformatting of all types ofrecords.

We can not ignore digital technology,it is not going to go away.

We must learn to:

•Be critical observers of technology.

•Honestly evaluate its strengths andweaknesses.

•Be sophisticated users of allappropriate tools.

Static Analog MediaPrinted publications, photographs,microfilm, and digital hardcopy output.

Dynamic Analog MediaMotion pictures, audio recordings, andvideo recordings.

Digital MediaAll types.

Static Analog MediaChemical and physical stability limits usablelife.

Dynamic Analog MediaChemical and physical stability and systemsobsolescence limit usable life.

Digital MediaSystems obsolescence is critical factor, butchemical and physical stability can beproblematic.

Digital

Motion Pictures

Video Recordings

Audio Recordings

Still Photographs

Textual Records

Increasingcomplexity

and

correspondingrisk of loss.

Digital Media

DynamicAnalog -machinedependent

Static Analog -human readable

Digital - digital preservation

Motion Pictures

Video - digital ref. only option

Audio - digital ref. only option

Still Photographs

Textual Records

Increasingexpensetoreformatand topreserve.

Harder to dopreservationreformatting asdigital - butonce digital,potentially noloss whencopying.

Feasible to dopreservationreformatting asdigital.

For preservation - use all of the following tools:

•Cost-benefit analyses

•Proper Storage / Cool and cold storage

•Environmental monitoring

•Holdings maintenance and conservation

•Risk, condition, and archival / curatorialassessment

•Reformatting

•Reformatting is almost always themost expensive option.

•For records that benefit from coldstorage and are not susceptible toformat obsolescence, we can storeoriginals in cold storage for between100 to 200 years for the same cost asreformatting everything.

•Reformatting remains the primaryapproach for preserving many systemdependent formats - particularlyaudio and video recordings.

•Reformatting remains essential forfacilitating access to all formats.

•Reformatting allows originals toremain in secure and/or cold storage.

Considerations for Reformatting:•Archival/curatorial considerations•Costs•Reproduction quality•Stability of imaging materials•Ease of distribution

•Fidelity

•Functionality

•Longevity•Source

characteristics

•Purpose/People

•Technologycapabilities

Original ResourcePerspective

From Steve Puglia’sReformatting pyramid chart

Institutional and UserPerspectiveFrom Steve Chapman’s

“Microfilm: A PreservationTechnology for the 21st Century?”

From Paul Conway’s“Overview: Rationale for

Digitization and Preservation”

•Use

•Sustainability

•Affordability

Digital reformatting isnot necessarily easier orless expensive thantraditional approaches toanalog reformatting.

•The amount of overall work iscomparable, digital is often slower, andthe technical complexity in many casesis greater.•For many types of originals,reformatting labs will need multipledigital workstations to achieve thesame capacity as traditional analog labequipment.

Digitization equipment andsupporting information technology(IT) infrastructure will have to bereplaced much more frequently.

•Analog lab equipment often had ausable life of 10 to 30 years, andsometimes longer.•Digital equipment will need to bereplaced every 3 to 5 years.

Analog is not going away any timesoon-

•We will have to maintain analoginfrastructure - particularly formachine-dependent formats - audio,video, and motion pictures.

•Not so easy as analog equipment isdiscontinued and ages - maintenance,parts, and repair will not be easy.

We will continue with analogreformatting in certain areas as longas possible-•Like motion pictures - because there aretechnical and economic limitations withdigital approaches.

•For motion pictures it is easy to convertto digital video at lower resolutions, buthard to do preservation reformatting.

We will have to incur majorreformatting expenses to copymachine-dependent formats in atimely manner.

It is not always possible orpractical to speed up the copyingprocess when the originals are inanalog format.

Need to take advantage of thebenefits of computers andinformation technology.

Need to develop comprehensiveworkflow tools and automate asmuch as possible to gainefficiencies in digitization on alarge scale.

Digitization is equal parts-•Digital conversion

•Description / cataloging /indexing and creation of othermetadata

•Project management

Systems Perspective:Managing and preservingdigital data/objects/records isdifferent than managing andpreserving physical records.

Digital•Everyone working to determine all that willbe necessary to preserve digital data over thelong term - we have made tremendousprogress over the last 5 to 10 years.

•Will need to preserve digitally createdmaterials using digital technology.

•Media reversion is not an option for manytypes of records, will lose functionality andthen it is no longer the record.

Need to move away frommanaging and preservingtechnology to approaches formanaging and preservingdigital objects/data.

Actively manage risk.

Need to bring digitalresources into a managedenvironment as soon aspossible, to facilitatemanagement, access, andlong-term preservation.

In order to consider usingdigitization as a method ofpreservation reformatting -It will be necessary to specifymore about the characteristicsand quality of the digitalcopies than what is in manycurrent digitization guidelines.

As an example - for raster images, needfor consensus on a variety of technicalfactors such as:

•Capture device performance•Tone reproduction and orientation•Color reproduction•Bit depth•Sampling frequency or spatial resolution•Other quality parameters•Image and signal processing procedures

•NARA’s 2004 Technical Guidelines forDigitizing Archival Materials for ElectronicAccess: Creation of Production Master Files -Raster Images may prove appropriate forcertain collections and institutions.•But - the decision to use the Guidelinesshould be based on a critical evaluation ofneeds, and not just be accepted because theGuidelines are available.

Other aspects beyond imagingparameters must be addressed inorder to be considered preservationreformatting.•Such as selection, metadata, rights,digital preservation, managedenvironment, etc.•Currently, the NARA Guidelines donot address these issues.

Why use digitization for preservationreformatting?•Appropriate analog processes and materials arenot or will not be available in the future forreformatting originals.

•We will have to use digitization for allpreservation reformatting, just as audiorecordings now can only be reformatted digitally.

•Digitization meets current needs for facilitatingaccess and limiting handling of originals.

Defining approaches forpreservation reformattingusing digitization -Started work two years ago ondefining levels of informationcapture for photographs.

•Fidelity

•Functionality

•Longevity

•Sourcecharacteristics

•Purpose/People

•Technologycapabilities

Original ResourcePerspective

Institutional and UserPerspective

CP Baseline CP IntermediateEP Baseline EP New Approaches

Less Assessment More Assessment

Highest risk fornot achievingpreservation goals.

Lowest risk for notachievingpreservation goals.

Fewer Uses Most Uses

CP Advanced

•Use

•Sustainability

•Affordability

Imaging EnvironmentUndefinedDefined

Image StateRAW

Prepped for aspecificoutput

OutputReferred -

looks towardsoutput

InputReferred -

looks towardssensor

Original Referred -defined relationshipbetween original and

digital version

CurrentPractice

EmergingPractice More technical

metadata isneeded

Should be ableto get by withless technical

metadata

Follow appropriate technicalguidelines and specifications,based on explicit decisionsrelating to specific needs.Need to collect all appropriatemetadata - documentingoriginals and digital objects.

Digitizing guidelines exist - stillwork to be done:

•Textual records - original paperrecords and microfilm•Still photographs•Audio recordings•People working on videorecordings and motion pictures

Need to define essential characteristicsof the original resources -

•Microfilm standards/guidelines focuson maintaining text legibility.•Specifications for photographicduplicates define approaches to produceduplicates that have the samephotographic properties as the originals-same overall density, density range, andrelationship between the tones.

Other Issues-•Quality control (QC) andquality assurance (QA).•Scanner/digital camera anddigitization equipmentevaluation and performance -limits and variability.

High-quality A/D converter (circa 2006):•Frequency response (1 Hz to 48 kHz): -1 dB•Total harmonic distortion and noise (1 kHz at 0 dBFS): < -108 dB(0.0005%) (unweighted RMS)•Dynamic range: > 130 dB (unweighted RMS)•Intermodulation distortion: <-90 dB•Spurious aharmonics: <-130 dBFS•Crosstalk (50 Hz, 0 dBFS in opposite channel): < -130 dB•Crosstalk (15 kHz, 0 dBFS in opposite channel): <-140 dB•Linearity (at -144 dBFS): < 3 dB•Intrinsic jitter: < 18 pSec RMS•Phase linearity: < 1｡•Internal clock accuracy: +5 ppm

“Measurement and Evaluation of Analog-to-Digital Converters Used in the Long-termPreservation of Audio Recordings” by Ken Pohlmann, www.clir.org/activities/details/AD-Converters-Pohlmann.pdf

For traditional reformatting,we developed infrastructureand defined workflows.Same for digital reformatting.

For digital reformatting, need atleast three tiers of infrastructure-•Highest - Digital Repository

•Intermediate - Shared Work Environment

•Lowest - Digital Reformatting Lab

At all levels - need to dothe routine IT things tomitigate risk of data loss.

It is about risk management.It is about creating a managedenvironment – the need to beproactive, not just reactive.

Effective IT procedures exist for the short-term management of electronic recordsand digital information-

• Not always followed.

• Not always as easy or as inexpensiveas advertised.

• We have been sold on the promise ofthe technology, but rarely acknowledgethe downsides.

Digital Reformatting LaboratoryThe local lab infrastructure -

•Not about managing and making informationavailable.•It is about creating digital objects from originalrecords of all types.•The digital equivalent of the traditionalreformatting lab.•Perform digitization, quality control, collecttechnical metadata, and in some cases performbaseline indexing.

At the lab level - need to define:•Baseline workflows to enhanceproductivity.•More specialized or customworkflows to address problems withoriginal resources and toaccommodate exhibits andpublications.

Shared Work Environment•Short-term place to make resourcesavailable to complete all other work.•Place where all additional work can bedone after the digital objects are created -such as additional QC, metadata creationand indexing, QC on metadata, etc.

Digital Repository•Long term management andpreservation of digital copies.•For NARA this will be theElectronic Records Archives(ERA).

The infrastructure will assist with-

Project / Reformatting Prep Work1. Manage workflow and coordinate preservation activitiesamong archival units, preservation labs, exhibits andpublications staff, web staff, and access staff.2. Prepare documents for digitization, both archival andpreservation/conservation preparation.3. Collect and record a more detailed level of descriptivemetadata during the course of archival prep work and toenhance the authoritative archival descriptions in officialdescriptive metadata repository.4. Document preservation/conservation actions inpreparation for digitization.

Reformatting Lab Work1. Perform digitization or analog (photographicduplicate) reformatting.2. Perform digitization using local workstations withspecialized equipment and software, and with local labnetwork including local digital storage capacity tofacilitate lab activities.3. Collect appropriate non-descriptive technicalmetadata resulting from the digitization process, likespecifications followed to produce digital resource.4. Perform preliminary organization of the digitalresources.

Post-Lab or Digitization Work1. Organize/structure digital resources based on theintellectual and physical organization of the records.2. Automated quality control (system checks) in additionto staff quality control (human checks) to identifyrework/defect correction.3. Rework/defect correction efforts.4. Track and associate new digital/analog versions ofrecords with the original records.5. Collect and prepare digital resources and metadataabout the resources for export into other systems.6. Provide for long-term digital storage for digital assetsand related metadata.

Creating-•Lots of choices available for digitizing equipment -even inexpensive office scanners can do a good job.•Scaling to meet large production demands,metadata creation/collection is a problem, andkeeping up quality (both resources and metadata) isa problem.•There are efficiencies when working from existingintermediates like microfilm, but may introducedeficiencies and may not capture all the desiredcharacteristics of the originals.

Creating-•While it seems easy to do, digitization stillrequires a lot of technical knowledge andskill•Can not ignore all the other things thatneed to be done - like metadata creation -good descriptive information is even morecritical for the accessibility of and long-termpreservation of digital copies.

Storage-•Moving away from individual removablestorage media (discs or tapes) to storage onspinning discs (servers with RAID).•Do not want to be managing physicalmedia - scalability of approach andmanagement is a problem.

Retrieving-•Metadata - standards and what is theminimum?•IT infrastructure - hardware and software.•Software - applications sold to manymarket segments that can be used.

Our need for software systems does notmatch a specific model or market segment,includes the following functionality-• Records management application (RMA)•Content management (CM)•Document capture and management• Digital asset management (DAM)•Media asset management (MAM)• Digital repository• Digital preservation

The cost for digital storagecontinues to drop, but themanaged environment (digitalrepository) may not be soaffordable - compared tostoring and managing human-readable and analog copies

Options-•Do everything locally - build andmanage the infrastructure yourselfvs.•Cooperative/consortial approachesvs.•Shared commercial options - hostedservices and application serviceproviders

Conclusions:•Start with getting the basics right,including hardware / softwareinfrastructure and metadata requirements.

•Finish defining appropriate approachesfor preservation reformatting usingdigitization.

•Use all appropriate tools - includingdigitization and new approaches fordigital preservation.