1

Presentation at SciDAC face-to-face

January 2005

Ron A. OldfieldSandia National Laboratories

The Lightweight File System

2

The Lightweight File System Project

• Participants:– SNL: Ward, Oldfield, Riesen, Lawry– UNM: Maccabe, Arunagiri

• Motivation– Risk-mitigation for Red Storm PFS (Lustre)– Vehicle for parallel I/O research

• Design Goals– Lightweight design

• only critical I/O functionality (direct access to storage, security)• special functionality implemented in I/O libraries (above LWFS)

– Scalable• expect clients with tens-to-hundreds of thousands of processes• I/O system should not hinder scalability of the application

– Secure• authentication and authorization• fine-grained access controls with revocation

An experimental object-based storage system(funded by Sandia’s CS research foundation)

3

Key features of LWFS

• Initial focus is on “secure storage architecture” (not a file system)• Policy vs mechanism

– Separate policy decisions from policy enforcement– Metadata servers and storage servers cooperate for policy consistency

• Access-control– capability-based – immediate revocation– automatic refresh – fine-grained operation control

• Containers for related objects– the unit for access control (no control for byte, object, or block access)– Every object is in a container– LWFS knows nothing about the structure of container– We have backwards pointers (not forwards)

4

What LWFS does not do

• naming (e.g, metadata)• data synchronization• data consistency• caching• prefetching• quota enforcement• data distribution

5

Basic LWFS Architecture

• Security model (Authentication and Authorization)– Separation of policy from mechanism– Scalable

• No connection-based mechanisms (e.g., Kerberos)

– Capabilities for access control (well... not quite)• not independently verifiable

• Similar to the NASD access credential

– Coarse-grained access control (containers)• Storage service

– object-based– enforcement of access control policies

6

Security Assumptions and Requirements

• We assume a trusted transport mechanism– prevents replay attacks, man-in-the-middle attacks, and

eavesdropping– provides network integrity– We do not need to provide encryption

• Authentication– Use existing mechanisms (e.g., Kerberos, GSS-API)– Scalable (no connection-based schemes)

• Authorization– Coarse grained access controls (at the container level)– Capabilities for scalability– “Immediate” revocation of capabilities

• Integrity– Make sure users cannot modify capabilities

7

Understanding the Scalability Requirements

Net I/O

Service

Users

File I/O (m)Compute (n)

/home

• Prohibit ops with O(n) communications• Prohibit data structures of size O(n)• Limit ops with O(m) comms to rare events

The I/O system should not hinder scalability of application

8

Motivating the “Open-Architecture” Model

• Application-specific APIs

– MPI-IO

– HDF5, PnetCDF

– ChemIO, SalvoIO

• Application control of policies

– Caching and prefetching

– Data distribution

– Synchronization

Applications should choose nearly all features of the I/O system

application

LWFS

access to datasets,app-specific APIs,caching, prefetching

high-levelI/O lib

acesss to (parallel) files,reliability, data distribution,consistency, synchronization

access to bytes (in objects), metadata management,security

traditionalPFS

low-levelI/O lib

9

Status

• APIs defined – Storage service

– Authentication and Authorization

– Naming service

– Transactions (Locks and Journals)

• Prototype implementations– Storage service

– Authorization service

– Naming service (in progress)

10

Current and Future Work

• Vehicle for parallel I/O research– Scalable metadata– Application-specific I/O libs and file systems

• data distribution, data consistency, caching, prefetching to match access patterns

• mobile app/lib code (e.g., active disk)– Lock-free synchronization schemes– Lightweight database (using LWFS)

• Framework for developing production-level parallel I/O systems