Upload
hugo-carpenter
View
233
Download
1
Embed Size (px)
Citation preview
Chap. 6 - Storage 2
Chapter Overview6.1 Introduction
6.2 Types of Storage Devices
6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.
6.5 Reliability, Availability and RAID
Chap. 6 - Storage 3
IntroductionThe Big Picture: Where are
We Now?
We will look at how devices (especially disks) are put together.
We’ll look at how to connect IO devices to the CPU.
And then we’ll look at RAID, the brainchild of Patterson and his buddies.
6.1 Introduction
6.2 Types of Storage Devices
6.3 Busses - Connecting IO Devices to CPU/Memory.
Interrupts etc. How is data transferred.
6.5 Reliability, Availability and RAID
Chap. 6 - Storage 6
Memory
I/O
The Processor Picture
Registers
Cache
Processor
Registers
Cache
Processor
Registers
Cache
Processor
Registers
Cache
Processor
Chap. 6 - Storage 7
Types of Storage Devices
In this section we will:
Take a quick look at how disks work. This is only one example of IO, but we will save networks, tapes, etc. for another course.
6.1 Introduction
6.2 Types of Storage Devices
6.3 Busses - Connecting IO Devices to CPU/Memory.
Interrupts etc. How is data transferred.
6.5 Reliability, Availability and RAID
8Chap. 6 - Storage
Disk Device TerminologyTypes of Storage Devices
• Purpose:– Long-term, nonvolatile storage– Large, inexpensive, slow level in the storage hierarchy
• Bus Interface:– IDE– SCSI – Small Computer System Interface
– Fibre Channel• Transfer rate
– About 120 Mbyte/second through the interface bus.
– About 5 Mbyte/second off of heads.– Data is moved in Blocks
• Capacity– Approaching 100 Gigabytes– Quadruples every 3 years
(aerodynamics)– Can be grouped together to get terabytes of data.
9Chap. 6 - Storage
Disk Device Terminology
Example: Seagate Cheetah ST336752LC
36 Gigabytes
15,000 RPM
3.6 ms avg seek time.
$699.00
Types of Storage Devices
http://www.seagate.com/cda/products/discsales/marketing/detail/0,1121,355,00.html
4 disks, 8 heads (so 8 tracks)
71,000,000 Total Sectors
18,000 cylinders
Average of 4,000 sectors/cylinder or 500 sectors / track (but different amounts on each track.)
MTBF = 1,200,000 hours
10Chap. 6 - Storage
Performance of Magnetic Disks
SectorTrack
Cylinder
HeadPlatter
15,000 RPM = 240 RPS => 4 ms per rev Average rotational latency = 2 ms500 sectors per track => 0.10 ms per sector512 bytes per sector => 5,000,000 MB / s
Response time = Queue + Controller + Seek + Rot + Xfer
Service time
Types of Storage Devices
Read Cache
Write Cache
Electronics (controller)
Data Control
Chap. 6 - Storage 11
Busses
In this section we will:
Look at various bus mechanisms.In very simple terms, a bus is the
connection between various chips/components in the computer.
The bus is responsible for sending data/control between these various components.
6.1 Introduction
6.2 Types of Storage Devices
6.3 Busses - Connecting IO Devices to CPU/Memory
6.4 I/O Performance Measures
6.5 Reliability, Availability and RAID
Chap. 6 - Storage 12
Interconnect Trends
Network
>1000 m
10 - 1000 Mb/s
high ( 1ms)
lowExtensive CRC
Channel
10 - 100 m
40 - 1000 Mb/s
medium
mediumByte Parity
Backplane
0.1 m
320 - 2000+ Mb/s
low (Nanosecs.)
highByte Parity
Distance
Bandwidth
Latency
Reliability
• Interconnect = glue that interfaces computer system components• High speed hardware interfaces + logical protocols• Networks, channels, backplanes
memory-mappedwide pathways
centralized arbitration
message-basednarrow pathwaysdistributed arbitration
Busses
Connects Machines ChipsDevices
Chap. 6 - Storage 13
A Computer System with One Bus: Backplane Bus
• A single bus (the backplane bus) is used for:
– Processor to memory communication
– Communication between I/O devices and memory
• Advantages: Simple and low cost
• Disadvantages: slow and the bus can become a major bottleneck
• Example: IBM PC - AT
Processor Memory
I/O Devices
Backplane Bus
Busses
Chap. 6 - Storage 14
A Two-Bus System
• I/O buses tap into the processor-memory bus via bus adaptors:– Processor-memory bus: mainly for processor-memory traffic– I/O buses: provide expansion slots for I/O devices
• Apple Macintosh-II– NuBus: Processor, memory, and a few selected I/O devices– SCCI Bus: the rest of the I/O devices
Processor Memory
I/OBus
Processor Memory Bus
BusAdaptor
BusAdaptor
BusAdaptor
I/OBus
I/OBus
Busses
Chap. 6 - Storage 15
A Three-Bus System
• A small number of backplane buses tap into the processor-memory bus
– Processor-memory bus is only used for processor-memory traffic
– I/O buses are connected to the backplane bus
• Advantage: loading on the processor bus is greatly reduced
Processor Memory
Processor Memory Bus
BusAdaptor
BusAdaptor
BusAdaptor
I/O BusBackplane Bus
I/O Bus
Busses
Chap. 6 - Storage 16
North/South Bridge architectures: separate busses
• Separate sets of pins for different functions– Memory bus – Caches– Graphics bus (for fast frame buffer)– I/O busses are connected to the backplane bus
• Advantage: – Busses can run at different speeds– Much less overall loading!
MemoryProcessor Memory Bus
BusAdaptor
BusAdaptor
I/O BusBackplane Bus
I/O Bus
“backsidecache”
Busses Processor
Director
Chap. 6 - Storage 17
Bunch of Wires
Physical / Mechanical Characteristics – the connectors
Electrical Specification
Timing and Signaling Specification
Transaction Protocol
What defines a bus?Busses
Chap. 6 - Storage 18
• Synchronous Bus:
– Includes a clock in the control lines
– A fixed protocol for communication that is relative to the clock
– Advantage: involves very little logic and can run very fast
– Disadvantages:
• Every device on the bus must run at the same clock rate
• To avoid clock skew, busses cannot be long if they are fast
• Asynchronous Bus:
– It is not clocked
– It can accommodate a wide range of devices
– It can be lengthened without worrying about clock skew
– It requires a handshaking protocol
Synchronous and Asynchronous BusBusses
Chap. 6 - Storage 19
° ° °Master Slave
Control LinesAddress LinesData Lines
Bus Master: has ability to control the bus, initiates transaction
Bus Slave: module activated by the transaction
Bus Communication Protocol: specification of sequence of events and timing requirements in transferring information.
Asynchronous Bus Transfers: control lines (req, ack) serve to orchestrate sequencing.
Synchronous Bus Transfers: sequence relative to common clock.
Busses So FarBusses
Chap. 6 - Storage 20
• One of the most important issues in bus design:– How is the bus reserved by a device that wishes to use it?
• Chaos is avoided by a master-slave arrangement:– Only the bus master can control access to the bus:
It initiates and controls all bus requests– A slave responds to read and write requests
• The simplest system:– Processor is the only bus master– All bus requests must be controlled by the processor– Major drawback: the processor is involved in every transaction
BusMaster
BusSlave
Control: Master initiates requests
Data can go either way
Arbitration: Obtaining Access to the Bus
Busses
Chap. 6 - Storage 21
The Daisy Chain Bus Arbitrations Scheme
• Advantage: simple• Disadvantages:
– Cannot assure fairness: A low-priority device may be locked out indefinitely
– The use of the daisy chain grant signal also limits the bus speed
BusArbiter
Device 1HighestPriority
Device NLowestPriority
Device 2
Grant Grant Grant
Release
Request
wired-OR
Busses
Order is:1. Request2. Grant3. Release.
Chap. 6 - Storage 22
• Even memory busses are more complex than this
– memory (slave) may take time to respond
– it may need to control data rate
Bus Request
Bus Grant
Cmd+AddrR/WAddress
Data1 Data2Data
Simple Synchronous ProtocolBusses
Clock
Chap. 6 - Storage 23
Address
Data
Read
Request
Acknowledge
Master Asserts Address
Master Asserts Data
Next Address
Write Transaction
t0 t1 t2 t3 t4 t5
t0 : Master has obtained control and asserts address, direction (not read), data. Waits a specified amount of time for slaves to decode target
t1: Master asserts request line
t2: Slave asserts ack, indicating data received
t3: Master releases req
t4: Slave releases ack
Asynchronous Handshake (4-phase)Busses
This is Fig. 6.11
Chap. 6 - Storage 24
Address
Data
Read
Req
Ack
Master Asserts Address Next Address
t0 t1 t2 t3 t4 t5
t0 : Master has obtained control and asserts address, direction, data
Waits a specified amount of time for slaves to decode target\
t1: Master asserts request line
t2: Slave asserts ack, indicating ready to transmit data
t3: Master releases req, data received
t4: Slave releases ack
Read Transaction
Slave Data
Busses
Chap. 6 - Storage 25
• All signals sampled on rising edge• Centralized Parallel Arbitration
– overlapped with previous transaction• All transfers are (unlimited) bursts• Address phase starts by asserting FRAME#• Next cycle “initiator” asserts cmd and address• Data transfers happen on when
– IRDY# asserted by master when ready to transfer data– TRDY# asserted by target when ready to transfer data– transfer when both asserted on rising edge
• FRAME# de-asserted when master intends to complete only one more data transfer
EXAMPLE: PCI Read/Write TransactionsBusses
Chap. 6 - Storage 26
– Turn-around cycle on any signal driven by more than one agent
EXAMPLE: PCI Read TransactionBusses
27Chap. 6 - Storage
How The CPU Talks To The IO
The interface consists of setting up the device with what operation is to be performed-
• Read or Write• Size of transfer• Location on device• Location in memory
Then triggering the device to start the operation
When operation complete, the device will interrupt.
Interfacing I/O To The Processor
I/O instructions (in,out) unique from memory access instructions.LDD R0,D,P <-- Load R0 with the contents found at device D, port P.
Device registers are mapped to look like regular memory:LD R0,Mem1 <-- Load R0 with the contents found at device D, port P.
This works because an initialization has correlated the device characteristics with location Mem1.
Chap. 6 - Storage 28
CPU
IOC
(1) Issuesinstructionto IOC
memory
(2)
(3)
Device to/from memorytransfers are controlledby the IOC directly.
OP Device Address
target devicewhere commands are
IOP looks in memory for commands
OP Addr Cnt Other
whatto do
whereto putdata
howmuch
specialrequests
(4) IOC interrupts CPU when done
Interfacing I/O To The Processor
How The CPU Talks To The IO
ROM
RAM
I/O
Virtual Memory Pointing at IO space.
29Chap. 6 - Storage
Memory Mapped I/O
ROM
RAM
I/O
Interfacing I/O To The Processor
Some physical addresses are set aside.There is no REAL memory at these addresses.Instead when the processor sees these addresses, it knows to aim the instruction at the IO processor.
30Chap. 6 - Storage
Transfer Method 1:Programmed I/O (Polling)
CPU
IOC
device
Memory
Is thedata
ready?
readdata
storedata
yesno
done? no
yes
busy wait loopnot an efficient
way to use the CPUunless the device
is very fast!
but checks for I/O completion can bedispersed amongcomputationallyintensive code
Interfacing I/O To The Processor
Chap. 6 - Storage 31
Device Interrupts
• An I/O interrupt is just like the exception handlers except:
– An I/O interrupt is asynchronous
– Further information needs to be conveyed
• An I/O interrupt is asynchronous with respect to instruction execution:
– I/O interrupt is not associated with any instruction
– I/O interrupt does not prevent any instruction from completion
• You can pick your own convenient point to take an interrupt
• I/O interrupt is more complicated than exception:
– Needs to convey the identity of the device generating the interrupt
– Interrupt requests can have different urgencies:
• Interrupt request needs to be prioritized
Interfacing I/O To The Processor
Chap. 6 - Storage 32
add $r1,$r2,$r3subi $r4,$r1,#4slli $r4,$r4,#2
Hiccup(!)
lw $r2,0($r4)lw $r3,4($r4)add $r2,$r2,$r3sw 8($r4),$r2
Raise priorityReenable All IntsSave registers
lw $r1,20($r0)lw $r2,0($r1)addi $r3,$r0,#5sw $r3,0($r1)
Restore registersClear current IntDisable All IntsRestore priorityRTI
Ext
ern
al I
nte
rru
pt
PC saved
Disable
All Ints
Superviso
r Mode
Restore PC
User Mode
“In
terr
up
t H
and
ler”
Device Interrupts
• Advantage:– User program progress is only halted during actual transfer
• Disadvantage, special hardware is needed to:– Cause an interrupt (I/O device)– Detect an interrupt (processor)– Save the proper states to resume after the interrupt (processor)
Interfacing I/O To The Processor
33Chap. 6 - Storage
Transfer Method 2:Interrupt Driven Data Transfer
CPU
IOC
device
Memory
addsubandornop
readstore...rti
memory
userprogram(1) I/O
interrupt
(2) save PC
(3) interruptservice addr
interruptserviceroutine(4)
Device xfer rate = 10 MBytes/sec => 0 .1 x 10-6 sec/byte => 0.1 µsec/byte => 1000 bytes = 100 µsec 1000 transfers x 100 µsecs = 100 ms = 0.1 CPU seconds
User program progress only halted during actual transfer. Interrupt handler code does the transfer.
1000 transfers at 1000 bytes each: 1000 interrupts @ 2 µsec per interrupt 1000 interrupt service @ 98 µsec each = 0.1 CPU seconds
Still far from device transfer rate! 1/2 in interrupt overhead
Interfacing I/O To The Processor
Chap. 6 - Storage 34
Delegating I/O Responsibility from the CPU: DMA
• Direct Memory Access (DMA):
– External to the CPU
– Act as a master on the bus
– Transfers blocks of data to or from memory without CPU intervention
CPU
IOC
device
Memory
CPU sends a starting address, direction, and length count to IOC. Then issues "start".
IOC provides handshakesignals for PeripheralController, and MemoryAddresses and handshakesignals for Memory.
Interfacing I/O To The Processor
35Chap. 6 - Storage
Transfer Method 3:Direct Memory Access
CPU
IOC
device
Memory
Time to do 1000 xfers at 1000 bytes each:
1 DMA set-up sequence @ 50 µsec1 interrupt @ 2 µsec1 interrupt service sequence @ 48 µsec
.0001 second of CPU time
CPU sends a starting address, direction, and length count to DMAC. Then issues "start".
IOC provides handshake signals for PeripheralController, and Memory Addresses and handshakesignals for Memory.
0ROM
RAM
Peripherals
IO Buffersn
Memory Mapped I/O
Interfacing I/O To The Processor
Chap. 6 - Storage 36
RAID
Redundant Array of Independent Disks
In this section we will:
• Motivate a need to have greater reliability and availability for disk data.
• Look at ways to get this greater reliability.
6.1 Introduction
6.2 Types of Storage Devices
6.3 Busses - Connecting IO Devices to CPU/Memory.
Interrupts etc. How is data transferred.
6.5 Reliability, Availability and RAID
37Chap. 6 - Storage
Array Reliability
• Reliability of N disks = Reliability of 1 Disk ÷ N
1,200,000 Hours ÷ 100 disks = 12,000 hours 1 year = 365 * 24 = 8700 hours
Disk system MTTF: Drops from 140 years to about 1.5 years!
• Arrays (without redundancy) too unreliable to be useful!
Hot spares support reconstruction in parallel with access: very high media availability can be achieved
Hot spares support reconstruction in parallel with access: very high media availability can be achieved
RAID
38Chap. 6 - Storage
Redundant Arrays of Disks
• Files are "striped" across multiple spindles• Redundancy yields high data availability
Disks will fail
Contents reconstructed from data redundantly stored in the array
Capacity penalty to store it
Bandwidth penalty to update
Mirroring/Shadowing (high capacity cost)
ParityTechniques:
RAID
39Chap. 6 - Storage
Redundant Arrays of DisksRAID 1: Disk Mirroring/Shadowing
• Each disk is fully duplicated onto its "shadow" Very high availability can be achieved
• Bandwidth sacrifice on write: Logical write = two physical writes
• Reads may be optimized
• Most expensive solution: 100% capacity overhead
Targeted for high I/O rate , high availability environments
recoverygroup
RAID
Probabliity of failure (assuming 24 hours MTTR) = 24 / ( 1.2 X 106 X 1.2 X 106 ) = 6.9 x 10-13 = 170,000,000 years
40Chap. 6 - Storage
Redundant Arrays of Disks RAID 3: Parity Disk
P100100111100110110010011
. . .
logical record 10010011
11001101
10010011
00110000
Striped physicalrecords
• Parity computed across recovery group to protect against hard disk failures 33% capacity cost for parity in this configuration wider arrays reduce capacity costs, decrease expected availability, increase reconstruction time• Arms logically synchronized, spindles rotationally synchronized logically a single high capacity, high transfer rate disk
Targeted for high bandwidth applications: Scientific, Image Processing
RAID
41Chap. 6 - Storage
Redundant Arrays of Disks RAID 5+: High I/O Rate Parity
A logical writebecomes fourphysical I/Os
Independent writespossible because ofinterleaved parity
Reed-SolomonCodes ("Q") forprotection duringreconstruction
A logical writebecomes fourphysical I/Os
Independent writespossible because ofinterleaved parity
Reed-SolomonCodes ("Q") forprotection duringreconstruction
D0 D1 D2 D3 P
D4 D5 D6 P D7
D8 D9 P D10 D11
D12 P D13 D14 D15
P D16 D17 D18 D19
D20 D21 D22 D23 P
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Disk Columns
IncreasingLogical
Disk Addresses
Stripe
StripeUnit
Targeted for mixedapplications
RAID
42Chap. 6 - Storage
Problems of Disk Arrays: Small Writes
D0 D1 D2 D3 PD0'
+
+
D0' D1 D2 D3 P'
newdata
olddata
old parity
XOR
XOR
(1. Read) (2. Read)
(3. Write) (4. Write)
RAID-5: Small Write Algorithm
1 Logical Write = 2 Physical Reads + 2 Physical Writes
RAID
43Chap. 6 - Storage
Subsystem Organization
hostarray
controller
single boarddisk
controller
single boarddisk
controller
single boarddisk
controller
single boarddisk
controller
hostadapter
manages interfaceto host, DMA
control, buffering,parity logic
physical devicecontrol
striping software off-loaded from host to array controller
no applications modifications
no reduction of host performance
RAID
Cache
44Chap. 6 - Storage
System Availability: Orthogonal RAIDs
ArrayController
StringController
StringController
StringController
StringController
StringController
StringController
. . .
. . .
. . .
. . .
. . .
. . .
Data Recovery Group: unit of data redundancy
Redundant Support Components: fans, power supplies, controller, cables
End to End Data Integrity: internal parity protected data paths
RAID
45Chap. 6 - Storage
System-Level Availability
Fully dual redundantI/O Controller I/O Controller
Cache &Array Controller
Cache & Array Controller
. . .
. . .
. . .
. . . . . .
.
.
.RecoveryGroup
Goal: No SinglePoints ofFailure
Goal: No SinglePoints ofFailure
host host
with duplicated paths, higher performance can beobtained when there are no failures
RAID
Chap. 6 - Storage 46
Summary6.1 Introduction
6.2 Types of Storage Devices
6.3 Busses - Connecting IO Devices to CPU/Memory.
Interrupts etc. How is data transferred.
6.5 Reliability, Availability and RAID
Chap. 6 - Storage 47
Course SummaryDuring this course, we’ve started to learn about the details of
computer architecture. Items included:
Instruction Sets - especially a glimpse at the Intel instruction set.
Pipelines - the gyrations necessary to speed up the processor.
Memory - the various elements in the hierarchy designed to speed up the effective access to data.
IO - a brief look at disks, busses, and how they are put together.