Embed Size (px)
Citation preview
YFS: An Introduction to the next /afs®
Jeffrey Altman, Daria Brashear, Marc Dionne, & Simon WilkinsonYour File System Inc. and Your File System Ltd.2014 European AFS and Kerberos Conference
Your File System Inc. (YFSI) is a New York State Corporation with HQ in Manhattan and registered as a business entity in Canada
Your File System Ltd is a wholly owned subsidiary of YFSI with HQ in London
YFSI is privately owned and operated
YFSI is a Red Hat Partner ISV
Your File System
Location Transparency: one name space User Mobility: access from any device Security: Flexible model for authentication, privacy,
data protection and access control Availability: Temporary loss to small groups for short
time periods Integrity: No user initiated backups Heterogeneity: Multiplatform Self service: Low Help Desk costs Atomic Publishing: Software, documentation, web
sites, .. Real time collaboration: Distributed File Locking Distributed Administration
The /afs® Vision
The vision was decades ahead of its time in 1983
The implementation is decades behind in 2014
AFS® is 30 years old
Limited network throughput Increased call processing latency Decreased service reliability and availability Elevated risk of distributed deadlocks Inability to use full capability of available hardware Failure to keep up with competing technologies
That /afs is still in use today is a credit to its vision and the strength of its architecture.
The Price of Inaction
Major system rewrites are few and far between
“Contractor Model of Support” leads to many small and localized changes
A lack of consistent vision and quality control Few incentives to invest in the next 30 years
13 Years of Open Source
The YFS Difference
Application Transparency• Be a Tier One file system on all major OSes
Embrace multi-producer, multi-consumer work flows
Extended Integrity: Disaster Recovery
The YFS Vision for /afs
Be performance competitive• Lustre, GPFS, Panasas, …
Best of breed data security Improved Ease of Use Designed for the long term
The YFS Vision for /afs
Improved performance with existing hardware Cost reductions due to hardware consolidation Zero data loss as part of a transition No flag day required
• Mixed deployments are encouraged
Preserving the old while providing the new
Performance
Performance issues restrict the jobs that sites are willing to run in /afs
Deploying excessive hardware to solve load distribution and fairness problems is expensive
Support for multiple file systems costs money, requires additional staff, can result in data duplication and out of sync issues
Performance: Why Does it Matter?
Reduced contention in the listener thread• 10 gbit network interface saturation
255 packet window size (per call) without degradation
• Order of magnitude faster on high latency links Dynamic Thread Pools
• Thread Count limited by OS resources
RX Performance
64-bit volume IDs 96-bit (79 octillion) vnode IDs 64-bit,100ns granular timestamps
• 2038 ready Ubik databases extensible up to 16 exabytes Partitions, volumes and quotas tracked up to
16 zettabytes
Scalability: Name Space Growth
Optimized Cache Manager handshakes Volume Status Information
• Reduces number of GetVolumeStatus RPCs• Permits RW / RO data cache sharing• Improved caching of RO volume per user permissions Fewer FetchStatus RPCs for RO volumes
Performance: Message Flow Optimizations
Host and callback package rewritten• Significantly faster callback breaks
Vnode lock contention dramatically reduced
Distributed writing to shared data sets now possible
Performance: Fileserver
Open mode supported on some OSes Bypasses VFS cache and AFS cache for both
read and write No file threshold to tune Data is copied directly to the caller, or directly
from the caller to the file server
Performance: POSIX O_DIRECT
Security
Data breaches and exposures are followed by a high cost
• Public Relations Nightmare• Costs of Identity Theft Detection Services (in U.S.)• Loss of employment for key staff members• Organizational reorganization• Disruption of core mission when forced to address
security concerns in crisis mode
Security: Why Does it Matter?
Multi-layered policies• Flexibility for self service end users• System administrator controls
Network Security Reduced Information Exposure Minimal Privilege Services
What is YFS Security?
Self Service Group Management Per-Object ACLs
• Cross directory hard links now permitted
Security: End Users
Volume ACLs• Limits the permissions that end users can grant
Security: System Administrators
Volume Security Policy• Per-Volume minimum acceptable rx connection
security properties File Server Security Policy
• Per-server minimum acceptable rx connection security properties
• Only volumes with weaker or equivalent security policy can be attached, moved to, or restored to.
Security: System Administrators
YFS RXGK Security Class• GSS Kerberos 5 authentication• AES-256 wire privacy and integrity protection
Cell wide key for DB servers Individual keys for file servers Per-host keys for BOS Overseer Service
Security: Network
YFS protects the callback channel with AES-256 privacy and integrity protection
• when rxgk is used for the incoming connection• Avoids leaking information about volume and file ids
accessed by a client• Prevents forged messages from invalidate callback
state
Security: Callbacks
Server Processes execute under a daemon account
• Not Root
Security: Minimal Privileges
Cache Managers can be issued• a Kerberos keytab • a Protection DB Machine ID
Keyed Cache Managers can use privacy for all connections
Machines IDs are similar to User IDs• Can be placed on ACLs and added to Groups• But are not included in system:authuser
Security: Keyed CMs & Machine IDs
New Capabilities
Per File ACLs Cross directory hard links
Extensions for Microsoft Windows Mandatory Locks
• Advisory locks are not enforced by the file server Symlink Updates
• Reparse Points can be updated without FileID change CreateFile with Lock
• Avoids races when simulating OpLock semantics
File System Extensions
• Modifications to human readable and machine readable output
• vos examine, listvol, rxdebug, xstat_fs, … Consolidate output Introduce consistency across command options
• Machine readable output –format is not human formatted All fields are now separated by single tabs Easy to import into spreadsheets and databases
Command Output Clarity
• All libraries are thread safe• Built using libtool• Intended for use implementing language bindings
Library Cleanup
• A library to obtain tokens rxkad yfs-rxgk
• aklog is a wrapper• Can be linked to pam modules
libacquire
Triggered by access denied errors Automatic Token acquisition using Logon
Session Kerberos Credentials Works with all applications that use
• WNet API: Network Providers• Shell API: Explorer, Office, anything with an Open
dialog
Automated Windows Domain Token Acquisition
Deployment and Configuration
Simplify Server Configuration Provide Extensibility for New Features BOS command lines are limited in length Permit the construction of flexible test suites
Why are Deployment and Configuration Important?
Greatly improved configuration flexibility and convenience
Custom file layouts are possible All settings centralized in a single configuration area,
single file or directory A configuration directory can ease distribution of
custom options All command line options can be set in configuration
Flexible Configuration
Goal Provide a test for every new feature, library function,
RPC Provide a test with every bug fix, if possible
Requirements Ability to spin up the various servers and provide a test
configuration All tests must be able to run as a regular user Must be able to serve test data not necessarily
under /vicep*
Test Suites
A complete test cell can spin up in a few seconds
Many tests spin up a cell and destroy it when done, maintaining test independence
Client testing through libafscp and fuse client All new features require tests before merging
Extensive test suite coverage
[Unit]Description=YFS Server ServiceAfter=syslog.target network.target
[Service]EnvironmentFile=-/etc/sysconfig/yfsExecStart=/usr/local/sbin/bosserver -config /s/yfs/server/yfs-server.conf -nofork
ExecStop=/usr/local/bin/bos -config /s/yfs/server/yfs-server.conf shutdown hurricane.marcdionne.net -wait -localauth
User=yfsGroup=yfs
[Install]WantedBy=multi-user.target
Sample systemd yfs-server.service
[marco@hurricane /s/yfs/server ]$ ls -ltotal 60drwxr-xr-x. 2 yfs yfs 4096 Mar 23 04:00 bos-rw-r--r--. 1 yfs yfs 526 Jul 15 2013 bos.keytab-rwxr-xr-x. 1 yfs yfs 26 Jul 15 2013 cacheinfodrwxrwxr-x. 6 yfs yfs 4096 Jan 11 15:36 datadrwxrwx---. 2 yfs yfs 4096 Oct 25 10:47 db-rw-r--r--. 1 yfs yfs 4 Jan 6 09:52 KeyFile-rw-r--r--. 1 yfs yfs 144 Jan 6 09:52 KeyFileExtdrwxrwx---. 2 yfs yfs 4096 Mar 25 10:28 localdrwxrwxrwx. 2 yfs yfs 12288 Mar 25 10:29 logs-rw-r--r--. 1 yfs yfs 15 Sep 12 2013 ThisCell-rw-r--r--. 1 yfs yfs 114 Dec 19 16:56 UserList-rw-r-----. 1 yfs yfs 2000 Aug 5 2013 vl.keytabdrwxrwxr-x. 2 yfs yfs 4096 Mar 26 18:25 yfs-server.conf
[marco@hurricane /s/yfs/server ]$ ls -l yfs-server.conf/total 8-rw-r--r--. 1 yfs yfs 645 Mar 26 18:25 cellservdb.conf-rw-rw-r--. 1 yfs yfs 792 Mar 4 15:48 yfs-server.conf
Sample file layout
[cells] example.com = { description = "Test cell" servers = { blizzard.marcdionne.net =
{ addr = 192.168.0.113 } } } marcdionne.net = { description = "Marc's cell" servers = { hurricane.marcdionne.net = { addr = 192.168.0.107 } } }
grand.central.org = { description = "GCO Public CellServDB 23 Apr
2008" servers = { penn.central.org = { addr = 128.2.203.61 } grand.mit.edu = { addr = 18.9.48.14 } andrew.e.kth.se = { addr = 130.237.48.87 } } }
Sample cellservdb.conf
[dirpath] SERVER_ETC_DIR = /s/yfs/server SERVER_DB_DIR = /s/yfs/server/db SERVER_LOGS_DIR = /s/yfs/server/logs SERVER_BOSCONFIG_DIR = /s/yfs/server/bos SERVER_LOCAL_DIR = /s/yfs/server/local SERVER_PART_PREFIX_DIR = /s/yfs/server/data
[fileserver] d = 125 p = 200 nojumbo = auditlog = /s/yfs/server/logs/audFile security = yfs-rxgk:crypt rxkad:clear rxnull
rxkad:crypt
[vlserver] keytab = /s/yfs/server/vl.keytab auditlog = /s/yfs/server/logs/audVl
[volserver] d = 125 auditlog = /s/yfs/server/logs/audVol
[bosserver] auditlog = /s/yfs/server/logs/audBos
[ptserver] auditlog = /s/yfs/server/logs/audPt
[salvager] auditlog =
/s/yfs/server/logs/audSalv
[salvageserver] auditlog =
/s/yfs/server/logs/audSalvserv
Sample server configuration
Packaging
Installation is the initial experience an end user has with the product
If the installation process is frustrating, the end user is likely to be unhappy with the product
Lack of digital signatures can block the installation of a package or trigger a frightening dialog
Why Packaging Changes are Important?
New installation packages• Windows• OSX• Linux Debian Fedora RHEL6 and RHEL7
Installation Packages
Microsoft Windows®
Single installer• 64-bit and 32-bit components• Heimdal Side by Side Assembly• Heimdal Command Line tools• Automatic Cache Sizing
All components digitally signed• Microsoft Cross Signing of Drivers
Microsoft Windows®
OSX
Flat package Integral packages for client, server and
development Digital signatures on the package, the kext
and the binaries using Apple-issued certificate
OSX
New packaging for Debian, Fedora and RHEL Integral packages for client, db services, and
file service Digital signatures on installation packages
Linux
Dual Protocol Stack
Allows advanced features while maintaining backwards compatibility with AFS®
AFS protocol suite has all of the capabilities and limitations of OpenAFS
YFS features only available on YFS protocol suite• rxgk, file server, vol server, vl server, pt server
enhancements Transparent negotiation of protocol suite
Dual Protocol Stacks
Two cell types are defined:• AFS cell deploys OpenAFS or IBM AFS vlservers• YFS cell deploys the YFS location server
OpenAFS and YFS File Servers can be joined to either cell
Mixed Mode Cells
No support for RXGK, AES-256 No support for file server security policies
YFS Client in AFS Cell
Improved RX Performance for writes No Rxgk Volume IDs restricted to AFS limits Security Policies cannot be enforced Only AFS compatible capabilities can be
registered IPv6 addresses cannot be registered
YFS Server in AFS Cell
YFS File Server in AFS cellAFSvlserver
AFSfileserver
YFSfileserver
AFS volumeformat
vos
Improved RX Performance for writes No Rxgk Volume IDs restricted to AFS limits Security Policies cannot be enforced Only AFS compatible capabilities can be
registered IPv6 addresses cannot be registered
YFS Server in AFS Cell
No support for RXGK, AES-256 No support for file server security policies Volumes with ID above 232-1 inaccessible Mandatory locks cannot be requested but will
be enforced Volume sizes and quotas >231KB will be faked Other restrictions as required to enforce
security policies
AFS Client in YFS Cell
AFS File Server in YFS cell
59
YFSlocationserver
AFSfileserver
YFSfileserver
AFS volumeformat
vos
YFS volumeformat
RW volumes on YFS server cannot be replicated to AFS server
Volumes containing YFS tags cannot be moved to an AFS server• ACL Data• Volume Attributes (ACL or Security Policy)
Data transfers protected with Rxkad and Fcrypt RX performance improved in YFS to AFS
direction
AFS and YFS Volserver Compatibility
61
YFS protocol suite AFS protocol suite
YFS POSIX attribute backend store
YFS File Server
YFS Protection Server
62
YFS protocol suite AFS protocol suite
64 bit Ubik database
YFS Location Server
63
YFS protocol suite AFS protocol suite
64 bit Ubik database
rxgkkeyserver
Documentation
Updated man pages New Quick Start Guides Updated Administrator’s Guide
Documentation
Licensing
The U.S. Government has classified YFS 1.x as a mass market product
Worldwide Export permitted with a few exceptions
No export restrictions on distribution by customers
Export Licenses
A full suite of clients and servers• Windows• OSX• iOS• RHEL5, RHEL6, RHEL7• Fedora• Debian• Solaris• AIX
YFS 1.0 Binary License
Free updates to new releases (one year)• Every four month release cadence
Free security updates (two years) Unlimited e-mail / web support (one year) Cell performance evaluation (once per year) Remote monitoring service (one year)
Support
Cell (no replication) 1 Server (DB and File)
Base cell (replication)4 DB Servers4 File Servers1000 User or Machine IDsUnlimited Client devices
Additional Servers
Additional IDs
Annual purchases continue supportNon-redistribution Source code license availableTraining (on-site or web)
Products
General Availability End of May 2014
First update, September 2014
Availability
Feature Priorities• IPv6 enhancements• Rapid Partition Relocation• Extended Volume Names• New Directory Format
Unlimited Directory Sizes Extended Attributes Alternate Data Streams
• Read/write Replication• Extended Callbacks
2014 Road Map
Questions! Answers?
