Coming revolutions in mass storage: Coming revolutions in mass storage:

implications for image archivesimplications for image archives Christopher D. Elvidge, Ph.D.NOAA-NESDIS National Geophysical Data Center E/GC2325 Broadway, Boulder, Colorado 80305 USAEmail: chris.elvidge@noaa.gov

And

Dr. Mikhail ZHIZHIN Head of Information Technologies Lab Institute of Physics of the Earth and Geophysical Center Russian Academy of Science, Moscow, RussiaEmail: jjn@wdcb.ru

APAN eScience Workshop – July 6, 2004

Image Archive Sizes Continuing to Grow RapidlyImage Archive Sizes Continuing to Grow Rapidly

• For example, from 1992-2004 satellite image ingest at For example, from 1992-2004 satellite image ingest at NOAA-NGDC runs from six to ten GB per day.NOAA-NGDC runs from six to ten GB per day.

• Once launched (~2007), NPP will produce about 2 TB of Once launched (~2007), NPP will produce about 2 TB of data per day, which NOAA will archive.data per day, which NOAA will archive.

• During the NPOESS era (2010-2020+) there will three During the NPOESS era (2010-2020+) there will three satellites, each producing 2 TB of data per day, with satellites, each producing 2 TB of data per day, with NOAA responsible for the archive.NOAA responsible for the archive.

• There are many other examples.There are many other examples.

Abridged History of StorageAbridged History of Storage((http://www.disk-tape-data-recovery.com/storage-history.htmhttp://www.disk-tape-data-recovery.com/storage-history.htm))

• Punch cards – back when snakes had legs.Punch cards – back when snakes had legs.• Ticker tape – faster than punch cards.Ticker tape – faster than punch cards.• Magnetic tape – invented by IBM in 1952.Magnetic tape – invented by IBM in 1952.• 1956 - IBM introduces the 305 RAMAC (Random 1956 - IBM introduces the 305 RAMAC (Random

Access Method for Accounting and Control), the Access Method for Accounting and Control), the first magnetic hard disk storage system. The first magnetic hard disk storage system. The RAMAC stored 5 megabytes (MB) of data, was RAMAC stored 5 megabytes (MB) of data, was the size of two large refrigerators and cost the size of two large refrigerators and cost $10,000 per MB; the device could store 5 million $10,000 per MB; the device could store 5 million characters of data on 50 disks, each 24 inches in characters of data on 50 disks, each 24 inches in diameter. Each disk could hold an equivalent of diameter. Each disk could hold an equivalent of 25,000 punch cards.25,000 punch cards.

RAMAC – RAMAC – the first hard drive – the first hard drive – 1956 1956

Abridged History of StorageAbridged History of Storage (http://www.columbia.edu/acis/history/media.html) (http://www.columbia.edu/acis/history/media.html)

9-track tapes – workhorse of image archives in the 1960’s-early 1990’s. 50 mb at 1600 bpi.

IBM MSS cartridge (1982)held 50 mb.

0.2 mb tape strip from IBM Data Cell (mid-1960’s)

Current Standard – Current Standard – Tape Library SystemTape Library System

• Used by NASA, Used by NASA, NOAA, USGS and NOAA, USGS and many others.many others.

• Tape is widely Tape is widely regarded as the regarded as the standard for at least standard for at least another ten years.another ten years.

Storage Technology 9310 robotic Storage Technology 9310 robotic tape silo, can hold 6000 IBM 3590 tape silo, can hold 6000 IBM 3590 tapes. At 20 GB each the silo can tapes. At 20 GB each the silo can hold ~300 TB.hold ~300 TB.Circa 1999.Circa 1999.

LTO Tape Growth Path Already PlannedLTO Tape Growth Path Already Planned(http://www.lto-technology.com/newsite/index.html)(http://www.lto-technology.com/newsite/index.html)

Currently Available

Alternative to Tape Library Systems:Alternative to Tape Library Systems:Use “Local” Hard Drives Instead of TapeUse “Local” Hard Drives Instead of Tape

• Approximate price parity between tape Approximate price parity between tape and hard drives.and hard drives.

• Allows faster access.Allows faster access.• Several design options (SAN, NAS).Several design options (SAN, NAS).• Hard drive capacity already in the 200 GB Hard drive capacity already in the 200 GB

range and has been projected to reach 20 range and has been projected to reach 20 TB.TB.

• Data may be more easily corrupted.Data may be more easily corrupted.

Alternative to Tape Library Systems:Alternative to Tape Library Systems:Use “Local” Hard Drives Instead of TapeUse “Local” Hard Drives Instead of Tape

• http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=43http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=43• http://www.firingsquad.com/hardware/building_budget_storage_server/http://www.firingsquad.com/hardware/building_budget_storage_server/• http://www.archive.org/web/petabox.phphttp://www.archive.org/web/petabox.php• http://nbd.sourceforge.net/http://nbd.sourceforge.net/• http://www.storage.ibm.com/software/virtualization/sfs/http://www.storage.ibm.com/software/virtualization/sfs/• http://www.microsoft.com/windowsserver2003/techinfo/overview/san.mspxhttp://www.microsoft.com/windowsserver2003/techinfo/overview/san.mspx• http://www.enterprisestorageforum.com/technology/features/article.php/947551http://www.enterprisestorageforum.com/technology/features/article.php/947551• http://www.enterprisestorageforum.com/technology/features/article.php/981191http://www.enterprisestorageforum.com/technology/features/article.php/981191• http://www.cse.ohio-state.edu/~jain/refs/san_refs.htmhttp://www.cse.ohio-state.edu/~jain/refs/san_refs.htm• http://www.brocade.com/san/pdf/whitepapers/SANvsNASWPFINAL3_01_01.pdf http://www.brocade.com/san/pdf/whitepapers/SANvsNASWPFINAL3_01_01.pdf

Alternative to Tape Library Systems:Alternative to Tape Library Systems:Use “GRID” Hard Drives Instead of TapeUse “GRID” Hard Drives Instead of Tape

• Approximate price parity between tape and hard drives.Approximate price parity between tape and hard drives.• Allows faster access.Allows faster access.• Several design options.Several design options.• Hard drive capacity already in the 200 GB range and has Hard drive capacity already in the 200 GB range and has

been projected to reach 20 TB.been projected to reach 20 TB.• Community ownership may lead to more collaborations?Community ownership may lead to more collaborations?• Data may be more easily corrupted.Data may be more easily corrupted.• Agencies may also choose to build stand alone archive Agencies may also choose to build stand alone archive

to ensure long term data preservation.to ensure long term data preservation.• See essay http://isec.pl/papers/juggling_with_packets.txtSee essay http://isec.pl/papers/juggling_with_packets.txt

Nano-Storage-Technology Still Nano-Storage-Technology Still EmergingEmerging

• Molecular-scale nanowire memory cells promises unprecedented Molecular-scale nanowire memory cells promises unprecedented data storage http://www.azonano.com/news_old.asp?data storage http://www.azonano.com/news_old.asp?newsID=122 newsID=122

• Big Blue says breakthrough means millipede may crawl out of labBig Blue says breakthrough means millipede may crawl out of lab http://www.smalltimes.com/document_display.cfm?http://www.smalltimes.com/document_display.cfm?section_id=53&document_id=7860section_id=53&document_id=7860

InPhase Promotional Video

Holographic Data Storage Still Emerging

Implementations of Nano and Implementations of Nano and Holographic Data StorageHolographic Data Storage

• TapeTape

• CD like disksCD like disks

• Hard drivesHard drives

Greater storage density – lower Greater storage density – lower costs – but implementation routes costs – but implementation routes likely to extend current forms.likely to extend current forms.

Vision of Future Image ArchivesVision of Future Image Archives

• Data easily accessed – readily processedData easily accessed – readily processed

• Combination of data from multiple sites / Combination of data from multiple sites / multiple sourcesmultiple sources

• Copies of source data and processing Copies of source data and processing tools kept on long term storage mediatools kept on long term storage media

Storage Options in Future Image ArchivesStorage Options in Future Image Archives

Raw Data, MetadataProcessing Code, Higher Level Products

Long termSurvivableStorage

A.K.A.DataVault

Network Storage

Tape LibrarySystems

Raw Data,Metadata,Processing Code

GRIDStorage

Working SubsetsOf Archive

Raw Data, Metadata, Processing Code, Higher Level Products, Experimental Products, Assessments

Storage Options in Future Image ArchivesStorage Options in Future Image Archives

Raw Data, MetadataProcessing Code, Higher Level Products

DataVault

Open StorageFacility

Raw Data,Metadata,Processing Code

WidelyHeldData

Raw Data, Metadata, Processing Code, Higher Level Products, Experimental Products, Assessments

Num

ber

of U

sers

Regional ResourcesRegional Resources

• Singapore Data Storage Institute: Agency Singapore Data Storage Institute: Agency for Science, Technology & Research, or for Science, Technology & Research, or A*STAR (then known as the National A*STAR (then known as the National Science & Technology Board) and the Science & Technology Board) and the National University of Singapore (NUS) National University of Singapore (NUS) http://www.dsi.a-star.edu.sg/research/spintronics.htmlhttp://www.dsi.a-star.edu.sg/research/spintronics.html

• Others?Others?

Conclusions - Advances in storage capacity Conclusions - Advances in storage capacity & reductions in cost will allow archive & reductions in cost will allow archive storage to diversify – with copies held to storage to diversify – with copies held to meet specific objectives:meet specific objectives:

• Widely distributed collections used in current projects.Widely distributed collections used in current projects.

• Tape and hard drive media to provide operational access Tape and hard drive media to provide operational access from data centers.from data centers.

• Long term “survivable” storage – two or more copies on Long term “survivable” storage – two or more copies on highly durable media to preserve data hundreds of years highly durable media to preserve data hundreds of years – ability to survive technological collapse – reengineering – ability to survive technological collapse – reengineering of read capacity.of read capacity.