View
45
Download
0
Category
Preview:
DESCRIPTION
A Smart Port Card Tutorial --- Software John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd. Software Overview. No MSR details included in this talk That will come in the MSR Tutorial SPC Specific Kernel Modifications Building a Kernel config compile - PowerPoint PPT Presentation
Citation preview
June 16, 2002 SPC Tutorial 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
A Smart Port Card
Tutorial---
Software
John DeHartWashington University
jdd@arl.wustl.eduhttp://www.arl.wustl.edu/~jdd
June 16, 2002 SPC Tutorial 2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Software Overview
• No MSR details included in this talk– That will come in the MSR Tutorial
• SPC Specific Kernel Modifications• Building a Kernel
– config– compile
• Loading Kernel onto SPC to execute it– download via APIC
• Initiating Kernel execution– boot loader
• Configuring Devices– APIC
June 16, 2002 SPC Tutorial 3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC Kernel Modifications
• Kernel Configuration File:– Memory Disk
– Serial Console
– Most device drivers unneeded (e.g no Ethernet…)
• Source Code Changes:– PCI Device Configuration: ./dev/pci/if_apic_pci.c
• APIC Specific configuration handled by driver
– Change some APIC defaults: ./dev/ic/apic.c• made switch port the default
– No HLT instruction: locore.s
– Kernel Memory Page Allocation: • ./arch/i386/include/pmap.new.h
– RTC: ./arch/i386/isa/clock.c, ./ufs/ffs/ffs_vfsops.c• We still had to remove some RTC code...
June 16, 2002 SPC Tutorial 4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Steps to an Executing Kernel on an SPC
• Configure Kernel– Start with a typical NetBSD kernel config file
– Add SPC specific option:• options SPC
– add memory disk (filesystem will reside in kernel!)• options MEMORY_DISK*
• options MINIROOTSIZE
– serial console port• options CONSDEVNAME
• options CONS_OVERRIDE
– remove “extra” devices (Ethernet, mouse, …)
June 16, 2002 SPC Tutorial 5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Steps to an Executing Kernel on an SPC
• Filesystem– we use a vnode disk device: vnd(4)
• disk device in a regular file
• pseudo-device vnd– must be included in your kernel where you are building the FS
– Does NOT need to be included in your kernel for the SPC
– build filesystem• dd(1): create the file
• vnconfig(8) - configure vnode pseudo disk as a regular file
• disklabel(8) - label it
• newfs(8) - construct a new file system
• mount(8)/umount(8): mount/unmount filesystem
– mknod(8) - make device files
– populate regular files from an existing system
June 16, 2002 SPC Tutorial 6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
… Steps to an Executing Kernel on an SPC
• Compile Kernel• Make a symbol only version of kernel for /netbsd
– ps, netstat, … use symbols from /netbsd
– real kernel with file system CANNOT reside in itself!
• Download kernel and File System separately– see following slides on AAL5download
• Or, Download as one file:– Add File System to Kernel
• mdsetimage(8)
– Create an image version of Kernel• sets up bss section and symbol table
– Kernel bss file is ready to download!
June 16, 2002 SPC Tutorial 7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
… Steps to an Executing Kernel on an SPC
• Reset SPC• Download Boot Loader
– via APIC control cells – loaded at physical memory location: 0x0
• Flip bit at location 0xFFE00 to start boot loader– configure NorthBridge (before kernel download!!)
• cache, memory size, memory type, timing, etc
– wait for kernel download to complete• looks for location 0x3FC to change from 0 to 1
– set up parameters on stack and jump to kernel• kernel located at 0x100000
• Download kernel bss file• Flip bit at location 0x3FC - kernel starts booting...
This is the OLD WAYPlease stop using download_APIC
It is slow and painful
June 16, 2002 SPC Tutorial 8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
AAL5_Download
• Replaces old download_APIC utility• Can “shotgun” download to <= 8 SPCs at once
– Puts exactly same kernel on all 8
– This works fine for MSR where we want same kernel
– May not be good for all uses
– “Shotgun” number controlled from command line
• Can download kernel and file system separately– no need to load file system into kernel before download
– still need to load symbol version of kernel into file system before downloading
• Takes about 1 minute for the whole download and boot process.
June 16, 2002 SPC Tutorial 9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
AAL5_Download: For SPC
• Implemented and runs in Stages:– Stage1 uses APIC Control cells to:
• reset the SPC(s)
• download Stage2 to run on the SPC(s)
– Stage2 (boot.out) runs on the SPC• runs SPC boot code to configure CPU and Northbridge
• configures APIC on SPC to use AAL5 descriptors
• sends pings back to CP, waiting for Stage3
• receives AAL5 frames from Stage3 and sends ACK/NACK
– Stage3 runs on PC• syncs with Stage2
• downloads the kernel or FS via 36KB AAL5 frames
• retransmits if any errors
June 16, 2002 SPC Tutorial 10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
AAL5_Download: For SPC-II
• We are still working out the final details– We are starting to test the first batch of SPC-IIs
• Implemented and runs in Stages:– Stage2 (boot.out) runs on the SPC-II
• Already loaded on the IDE Disk on Module.– no need to download it
• runs SPC boot code to configure CPU and Northbridge• configures APIC on SPC to use AAL5 descriptors• sends pings back to CP, waiting for Stage3• receives AAL5 frames from Stage3 and sends ACK/NACK
– Stage3 runs on PC• syncs with Stage2• downloads the kernel or FS via 36KB AAL5 frames• retransmits if any errors
June 16, 2002 SPC Tutorial 11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Simple View
VCI=21
PC SPC-I
VCI=25
VCI=0x321
VCI=21
VCI=23VCI=24
Stage1
Stage1 APIC
Stage3 Stage2
June 16, 2002 SPC Tutorial 12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Sequential View
PC
Stage1
Stage3
SPC
APIC
Stage2
0x0Boot andStage2
0x0100000
Kernel andFilesystem
0x0FFE00
Boot ROM0xFFFFFFF0
0 0 0 0 0 0 0 00 0 0 0 0 0 0 1
SPC Memory
Controlcells
AAL5Frames
RESET
Resetcell
June 16, 2002 SPC Tutorial 13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Source and Executables
• source location: wu_arl/utilities/SPC/AAL5Download
• binary location:– wu_arl/utilities/bin/$OSTYPE/Stage1
– wu_arl/utilities/bin/$OSTYPE/Stage3
– wu_arl/utilities/bin/$OSTYPE/AAL5_download.sh
– wu_arl/utilities/bin/$OSTYPE/boot.out
– wu_arl/utilities/bin/$OSTYPE/FindNM
June 16, 2002 SPC Tutorial 14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Usage
• Individual command line commands– ./Stage1 –shotgun 8 –svci 21 –rvci 21 –dvci 23 –uvci 24 –cvci 25
– ./Stage3 –shotgun 8 –native netbsd.MSR –both 24
– ./Stage3 –shotgun 8 –bss SPC24MB.fs –address 0x0196434 –both 24 -go
• AAL5download.sh script– ./AAL5download 8 netbsd.MSR SCP24MB.fs 21 21 23 24 25
• Use from MSR Configuration Module– system(msrDownload.sh);
June 16, 2002 SPC Tutorial 15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Special Usage Note on Linux
• AAL5download.sh script:– needs to be able to determine the start location for the
File System
– NetBSD uses nm(1)
– Linux nm(1) won’t work on a NetBSD binary
– We have written a version of nm specifically for AAL5_download.sh: FindNM
– FindNM is used automatically when AAL5_download.sh is run on Linux
June 16, 2002 SPC Tutorial 16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
AAL5_download.sh Usage• > ./AAL5_download.sh
• Usage: ./AAL5_download.sh <netbsd_file> <filesystem_file> <start_vci> [<second_vci>] [<third_vci>][<fourth_vci>] [<fifth_vci>]
June 16, 2002 SPC Tutorial 17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Stage1 Usage• > ./Stage1• ./Stage1: [vpi/vci data] [load info]• ./Stage1: [-svpi #] (VPI for local sending of AAL5 control cells (0))• ./Stage1: [-svci #] (VCI for local sending of AAL5 control cells)• ./Stage1: [-rvpi #] (VPI for local reading of AAL5 control cells (0))• ./Stage1: [-rvci #] (VCI for local reading of AAL5 control cells)• ./Stage1: [-cvpi #] (VPI for remote CC writes of AAL5 control cells (0))• ./Stage1: [-cvci #] (VCI for remote CC writes of AAL5 control cells)• ./Stage1: [-dvpi #] (VPI for local write of AAL5 data frames (0))• ./Stage1: [-dvci #] (VCI for local write of AAL5 data frames)• ./Stage1: [-uvpi #] (VPI for local read of AAL5 data frames (0))• ./Stage1: [-uvci #] (VCI for local read of AAL5 data frames)• ./Stage1: [-resp #] (Response port of AAL5 control cells)• ./Stage1: [-start #] (Initial VCI sequence (41, 42, 43, 44))• ./Stage1: [-watch 0x#] (Monitor location for changes)• ./Stage1: [-apic #] (Linux: Apic Id (0 <= X <= 3))• ./Stage1: [-abit #] (Alternating bit setting)• ./Stage1: [-file <path>] (Name of boot loader (boot.out))• ./Stage1: [-swap] (use htonl() on output)• ./Stage1: [-shotgun #] (multicast loading 8X)• ./Stage1: [-msr #] (VCI for MSR ONLY AAL5 data frames)• ./Stage1: [-address #] (Physical data write address (0))• ./Stage1: [-noreset] (do not write reset register)
June 16, 2002 SPC Tutorial 18WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Stage3 Usage• > ./Stage3 -help
• [-switches] (-help) args
• -bss <file> Load .bss format file
• -native <file> Load native uncompressed file
• -strip Remove Symbols from kernel
• -debug print lots of stuff
• -verbose Dump Elf headers
• -elf Elf format execution
• -rvpi <VPI> Load VPI number
• -rvci <VCI> Load VCI number
• -svpi <VPI> Load VCI number
• -svci <VCI> Load VCI number
• -both <VCI> Set both VCI numbers
• -address <PhysAddr> Load file at <addr>
• -start <PhysAddr> Start file at <addr>
• -go Start kernel
• -shotgun <count> Multicast loading 8X
June 16, 2002 SPC Tutorial 19WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Host APIC to SPC Control Connections
APIC
PC
WUGS Switch
Port 0Port 1
Port 7
Port 6
Port 3Port 5
SPC/Port 4
Port 2
0/42
0/41
0/33 (0x21)
0/0x0321APIC ID
41 0x0321
Example
June 16, 2002 SPC Tutorial 20WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
APIC Driver at SPC Boot Time
• PCI Configuration Space (sys/dev/pci/if_apic_pci.c):
– Base memory address:• APIC Physical memory in SPC starts at 0xF0000000
– Memory mapping options:• Memory Enable• Master Enable• Write Invalidate Enable
– IRQ• Driver normally would read this from PCI config space• APIC statically assigned IRQ 5 by System FPGA• For consistency, driver writes 5 to Interrupt Line Register
June 16, 2002 SPC Tutorial 21WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
PCI Configuration Space
Configuration Space Header
Device ID Vendor IDStatus Command
Class Code Revision ID
BIST HeaderType
Cardbus CIS PointerSubsystem ID Subsystem Vendor ID
Expansion ROM Base AddressReservedReserved
Max_Lat Min_Gnt
LatencyTimer
Cache LineSize
Base Address Registers
InterruptPin
InterruptLine
June 16, 2002 SPC Tutorial 22WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Defining a Route on APIC Network Interface• NOTE: With the MSR you do not use ifconfig and atm_ifconfig
– the MSR uses a different APIC driver then “plain” NetBSD for the SPC
• Configure the APIC> ifconfig apic0 inet 192.168.10.1 netmask 0xffffff00
- network interface: apic0- our IP address: 192.168.10.1
• Configure a VCI> atm_ifconfig apic0 0xc5 open -aal5 -llc -besteffort -
maxqueuebufs 100 -inport FIBER -outports FIBER- network interface: apic0- vci: 0xc5- besteffort (vs. paced or lowdelay)- Tx buffer queue limit for this VCI: ‘maxqueuebufs 100’
• Configure the route> route add -iface 192.168.10.5 -link apic0:0.0.0.c5
- destination: 192.168.10.5- network interface: apic0- vci: 0xc5
June 16, 2002 SPC Tutorial 23WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
atm_ifconfig> atm_ifconfigUsage: atm_ifconfig <interface> <vpivci> [open|close|modify|getall [options...]]
Options are:
-aal0 | -aal5
-llc | -nollc <protocol>: use LLC/SNAP?
protocol is inet | ratm | value of ethernet type
-loopback
-inport <input_port_number>
-outports <bitvector_of_output_ports>
bitvector: 1 => port 0, 2 => port 1, 3 => both
-lowdelay | -paced <rate> | -besteffort
-maxqueuebufs <max_descriptors_in_tx_queue>
<rate> is in Kbits/sec
ChangesHERE
-inport and-outportvalueshave
changed.
June 16, 2002 SPC Tutorial 24WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
SPC
TI
Switch Fabric
IPP
OP
P
SPC
TI
IPP
OP
P
TI
ControlProcessor
APIC Configuration
6 56 5 6 5
IPP
OP
P
SPC
TI
IPP
OP
P
SPC
TI
. . .
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
33 3 66 6
SPC
June 16, 2002 SPC Tutorial 25WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Link
Defining APIC Ports for Routing
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
Switch
INPORT OUTPORTs
LINK LINK
June 16, 2002 SPC Tutorial 26WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Link
Defining APIC Ports for RoutingSwitch
INPORT OUTPORTs
LINK SWITCH
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
June 16, 2002 SPC Tutorial 27WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Link
Defining APIC Ports for RoutingSwitch
INPORT OUTPORTs
SWITCH LINK
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
June 16, 2002 SPC Tutorial 28WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Link
Defining APIC Ports for RoutingSwitch
INPORT OUTPORTs
SWITCH SWITCH
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
June 16, 2002 SPC Tutorial 29WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Link
Defining APIC Ports for RoutingSwitch
INPORT OUTPORTs
LINK BOTH
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
June 16, 2002 SPC Tutorial 30WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
IPP
OP
P
TI
Sys.FPGA
32-64MB
Pentium
Cache
NorthBridgeAPIC
Link
Defining APIC Ports for RoutingSwitch
INPORT OUTPORTs
SWITCH BOTH
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs
June 16, 2002 SPC Tutorial 31WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Caveats
• You cannot change the SPC-I IRQ assignments:– They are fixed in the System FPGA
– And there should be no reason to change them
• HLT instruction in locore.s commented out– remnant of when we were having INTEL problems
– we can leave it this way
– gives a slight performance improvement
• SPC needs to be reset before EACH download
Recommended