Internet2 Distributed Storage Infrastructure (I2-DSI)

TERENA-NORDUnet Networking Conference 1999

7-10 June, Lund, Sweden

Leif Abrahamsson, Ellemtel

leif.abrahamsson@ellemtel.se

UCAID/Internet2

Re-create leading edge R&E network capability

Enable new generation of applications

Integrate with production environment

Internet2 Goals

Internet2 Working Groups

IPv6MeasurementMulticastNetwork ManagementNetwork Storage

Quality of ServiceRoutingSecurityTopology

Internet2 Distributed Storage Infrastructure Update

Micah BeckUniv. of Tennessee, Knoxville

Bert DempseyUniv. of North Carolina, Chapel Hill

http://dsi.internet2.edu

I2-DSI Technical Participants

UT Knoxville / ICL M. Beck T. Moore

M. Swany J. Talley

UNC Chapel Hill /SILS B. Dempsey Paul. Jones

D. Weiss Z. Xiao

Site Managers NC Supercomputing Center Indiana University EROS Data Center Univ. of Hawaii at Manoa

Ellemtel L. Abrahamsson C. Achouiantz P. Johansson

IBM R Aiken G. Carpenter G. Goldszmidt J. Warren

Sun Microsystems R. Chawla (Healtheon)

Viagenie (with CANARIE) M. Blanchet F. Parent

A Word From Our Sponsors

Cisco DNS redirection Ellemtel engineering effort IBM large storage & DCE serversNovell storage & directory serversStarburst reliable multicast softwareStorageTek large storage serversSun design collaboration

Single Server Model

High performance locally

Unacceptable performance across commodity backbone

Relying on Wide Area QoS

High performance access with reserved bandwidth

Essential for real-time communication

Technically difficult, expensive, not generally available

I2-DSI Model: Replicated Services

Clients access nearby server

Everyone gets performance

Local resources implement a global service

I2-DSI Service Architecture

Replication

Rsync+, Omnicast, AFS/DFS

Novell Replication

Resolution

Sonar DNS, Distributed

Director

Delegation

Cache prefetch

generalusers

replicated core

delegated server

local users

Internet Content Channels

A channel is a collection of content which can be transparently delivered to end user communities at a chosen (price,performance) point through a flexible, policy-based application of resources

Server Channel Examples

Replicated Web Servers APIs: Standard HTML, Active Server Pages Channels: Web sites

Streaming Media APIs: MPEG-2, proprietary file formats Channels: collections of multimedia presentations

Executable content APIs: Java byte code, Tcl, Perl Channels: CGI programs

Current Server Deployment

I2-DSI Server

IBM RS/6000 AIX Server2 GB RAM72 GB Disk / 900 GB TapeADSM Hierarchical Storage Mgt.

I2-DSI Server Operations

Project Operations Coordinator Judi Talley, University of Tennessee at Knoxville

Site Managers Dave Vernon, Indiana University David Lassner, University of Hawaii at Manoa Mark Johnson, NC Networking Initiative Michael Rechtenbaugh, EROS Data Center

Infrastructure Expansion

StorageTek 2 PC/Linux Servers 700GB disk, tape backup

Novell 6 PC/NetWare Servers 100GB disk Smaller institutions or departments

InfrastructureDevelopment Efforts

Proximity ResolutionMartin Swany: SonarDNSGeoff Carpenter, German Goldszmidt: Narwhal (IBM)

Replication Mechanisms and Modeling Bert Dempsey & students:

Debra Weiss: Batch rsync + multicastZhiwei Xiao: Network metrics and modeling

Channel Representation and ServerLeif Abrahamsson, Christophe Achouiantz,

Patrik Johansson (Ellemtel)

I2-DSI Applications Workshop Chapel Hill, NC March 4 &5, 1999

10 applications Indiana: Digital music and

media library UNC-CH: Instructional

Management System San Jose State: Art history

images Vanderbilt: zoomable medical

images Viagenie: Network docs

database

Columbia: Earth sciences environment

UNC-CH: Virtual Laboratories

Ohio Supercomputer Center: High Volume Datasets

CalTech: Globally Interconnected Databases

Univ. of Kent: National Software Archive

Red Hat: pan-Linux source distribution

I2-DSI Applications Workshop Chapel Hill, NC March 4 &5, 1999

4 technologiesMinnesota: Scalable Video IBM Research: Multicast, Filter and StoreMoscow Ctr. for New Info. Tech. in Med. Ed.:

Semantic Text AnalysisIBM Research: Narwhal Resolution Proxy

http://dsi.internet2.edu/apps99.html

Special issue of the Journal of Network and Computer Applications (Academic Press)

Application Management Partner MetaLab.unc.edu

The site formerly known as SunSITE.unc.eduFearless Leader Paul Jones“A cool, tall glass of sweet tea on a hot day.” 2 M HTTP & 1/3 M FTP file transfers dailyCollections policy

teaching, research, or public service use technology in innovative and unique ways non-commercial or not-for-profit

Application Strategy

Chose initial applications Available or easily ported services Low update demands

Port to an I2-DSI server Our development effort is limited App developers can have access to the servers

Distribute to homogeneous coreDerive service abstractions

The Need for Channel Representation Standards

Origin Server

proxy

Replicated Server

Standard-based Web traffic

Replication of source files

Web clients

locally interpreted files

Replicated Server

replicated files

Replication Performance and Scalability Issues

Server placementServer resourcesServer description (metadata)Server Channel description (metadata)Object representationCharacterization of replication mechanismsChannel-to-server mapping (subscription)

Network Storage Symposium (NetStore ‘99)

Network Storage Symposium Seattle, Oct 14-15, 1999 http://dsi.internet2.edu/netstore99

Scope (look for CFP) I2-DSI implementation I2-DSI applications Related networking projects Storage technology

Conclusions

A server platform is in placeInfrastructure development

Service abstractions (search, computation) Publication and replication protocols Portable representation and API Heterogeneous servers

Six months to show results from initial application development efforts