View
7
Download
0
Category
Preview:
Citation preview
Open Source firmware a utoma tized tes ting
EDGE COMPUTING
Jean-Marie VerdunCTO ITRenew
• Leading provider of circular data center solutions to the hyperscale cloud market: technology, logistics, monetization
• Leading global provider of technology -enabled data center decommissioning services, including data sanitization software and IT asset remarketing
• Manufacturer of Sesame open compute and storage solutions, 1st of its kind suite of recertified and warrantied solutions for global use cases
• Delivering TCO value recovery > $1Billion to date
Redefining and powering LTV (a.k.a. not your grandfather’s ITAD)
Clients We Partner With
Project goals and genesis
• Build a full open source s ta ck from ha rdw a re to opera ting sys tem
• Provide ba ck end infra s tructure w hich could va lida te every pa tches a nd commits to the follow ing project by running a utoma ted tes tings
- linuxboot
- OpenBMC
Main principles• CI must be used from a
W eb API. Job Controller ta kes a s a n input a job description file w hich ca n s top the build a t a ny 4 predefined s ta ge.
• Output is a s ta tus a nd a rom file for a specific boa rd
• Job s ta tus ca n be queried through the API
job controller
builder
node brick va lida tion
ba s ic fea tures va lida tion
O/S s ta bility a nd s tress tes ts
W eb API
Active dev
Implementation1. Automa tic redeployment ma na ged through Ansible scripting2. Automa tic redunda ncy a nd sca la bility ba sed on ha rdw a re a va ila bility3. Slurm ba tch scheduler to ma na ge run queue, unique job ID, a nd jobs output4. KVM used to sa nd box builds into Ubuntu Xenia l VMs5. Ba s ic setup requires :
b. An a ns ible ma ster nodec. A Slurm controller noded. A Slurm ba tch node e . All of them sha ring the sa me subnet
Web API
GOALS1. Get a s a n input a g ithub
repository a ddress w ith a unique commit ID
2. Provide job control- La unch- Kill- Lis t
3 . Provide jobs s ta tus feedba ck- Build log file
INITIAL IMPLEMENTATION• W ritten in Go• Support
- Job la unch- Job s ta tus query- Job log
Job controller
GOALS1. Alloca te a nd ma na ge build nodes resources2. Preset build environment3. Store jobs s ta tus 4. Controlled us ing a hidden file w hich conta ins jobs description tha t w ill
override defa ult va lues
Job controller
INITIAL IMPLEMENTATION1. Alloca te job through Slurm ba tch scheduler2. Setup a virtua l ma chine (ba sed on Ubuntu Xenia l) w hen node is a lloca ted
w ith predefined cha ra cteris tics us ing KVM a nd virsh a. VM storage is seating in memory (about 40 GB)b. VM have access to the Internet and can run apt command to setup build
environment
3. Setup remote a ccess to the VM from the s lurm compute node4. Copy the re leva nt files into the VM a nd setup build environment5. Initia te job execution
BuilderGOAL• Build a fully functiona l linuxboot ima ge ba sed on job pa ra meters
INITIAL IMPLEMENTATION Ba sed on osresea rch/hea ds build environment
• Requires initia l boa rd ROM• Ca n build the linuxboot kernel (currently ba sed on 4.9 but ca n be cha nged)• Ca n a pply va rious pa tches to the kernel (BDS, AHCI etc …) ba sed on boa rd descriptor• Ca n build NERF (go ba sed) user environment (w ith full integra ted comma nd control) a s w ell
a s Hea ds (busybox/libc ba sed)• Ca n control/extra ct fina l DXE drivers integra ted w ithin linuxboot ROM• Genera te fina l linuxboot rom • Hea vy Ma kefile usa ge (looks like to be a “show stopper” to ma ny Z genera tion engineers)
Node brick validation
GOAL• Va lida te tha t a new ly built ima ge doesn’t brick a node (a ka tha t w e ca n ta lk
to the firmw a re through seria l a nd successfully execute ba s ic comma nd)
INITIAL IMPLEMENTATION• Ba sed on initia l qemu la unch of the ROM• Ba sed on rea l ha rdw a re setup w ith a ROM emula tor connected to the boa rd
Basic features validation
GOALS• Va lida te tha t a new ly built ima ge is properly detecting ha rdw a re• Va lida te tha t a new ly built ima ge is a ble to ins ta ll a n O/S a nd boot it through
- PXEboot or a ny other netw ork boot ca pa bility- loca l boot on AHCI a nd NVMe stora ge
Basic features validation
INITIAL IMPLEMENTATIONO/S ima ge ins ta ller built for linuxboot• Ubuntu Xenia l netins ta ller or loca l boot kernel fa ils to boot properly on
linuxboot (the kernel ha ngs)i. W e regenera te a full ISO ima ge ba sed a va lid origina l Ubuntu Xenia l ISO a nd
disa ble EFI support w ithin the ins ta ller kernel
ii. Tha t ima ge is a lso pre-configuring console output e ither to ttyS0 (loca l seria l) or ttyS4 (SoL) on W interfell ma chine
iii. Tha t new ima ge is a lso boota ble through a PXEboot process w hich is a utoma tica lly configured on the s lurm controller node.
O/S stability and stress testGOALS
Va lida te tha t a new ly built ima ge is a ble to:Run Linux a t full opera tiona l mode
i. Properly detect ha rdw a re beha vior (ACPI, CPU Frequencies ma na gement, Pow er Ma na gement, NUMA topology, VMX)
ii. Detect a nd ca n ma na ge w ithout error PCIe subsystems1. High speed netw ork a da pter (10 to 100 Gpbs)2. High speed s tora ge box (lightning)3. Low speed s tora ge box (knox)
iii. Can run various workload without system error and within an acceptable performance goal1. Run the Linpa ck/pysthone benchma rk a t speed2. Run Netw orking benchma rk a t speed3. Run bonnie++ a t speed4. Run Multiple VM implementa tions of the previous benchma rks
Using the CI as of today
Launching a build• Setup into a g it repo a .ci.yml file w hich conta ins 2 ba s ics entries
• Execute a protected curl reques t to http://<my_ci_server>:1234/v1/jobs getting a repo a nd bra nch to run from a s input pa ra meters
script -> Drive the execution w orkflow
a rtifa cts -> Expected output files
Using the CI as of today
Controlling executioncurl reques t to http://<my_ci_server>:1234/v1/jobs /<job_id>
Monitoring outputcurl reques t to http://<my_ci_server>:1234/v1/jobs /<job_id>/logs?ra w =true
Retrieving results - setting up complex inputThe build script sha ll ta ke ca re a bout pos ting re leva nt output results into a n end user repos itory or ftp build environment. Input like orig ina l ROM file sha ll be dow nloa ded by the build script into the re leva nt repos itory before build is kicked off.
What is next: PRC lab
• Live infra s tructure ba sed in Ca lifornia , Open to the community to tes t a nd va lida te firmw a re build
• Hosted by ITRenew• Dua l ra cks ba sed on OCP W interfell/Leopa rd servers
Call to Action
• Join ITRenew Open Ha rdw a re la b in s ilicon va lley (upcoming meetups)• Contribute to Open Source Firmw a re initia tive • Provide ha rdw a re for tes ting a nd build ea rly prototype• Conta ct us if you a re w illing to build a da pted des igns!• Dow nloa d des ign files on github!
• W a nt to commit code ? https ://g ithub.com/linuxboot/linuxboot-ci
Recommended