Open Source firmware a utoma tized tes ting

EDGE COMPUTING

Jean-Marie VerdunCTO ITRenew

• Leading provider of circular data center solutions to the hyperscale cloud market: technology, logistics, monetization

• Leading global provider of technology -enabled data center decommissioning services, including data sanitization software and IT asset remarketing

• Manufacturer of Sesame open compute and storage solutions, 1st of its kind suite of recertified and warrantied solutions for global use cases

• Delivering TCO value recovery > $1Billion to date

Redefining and powering LTV (a.k.a. not your grandfather’s ITAD)

Clients We Partner With

Project goals and genesis

• Build a full open source s ta ck from ha rdw a re to opera ting sys tem

• Provide ba ck end infra s tructure w hich could va lida te every pa tches a nd commits to the follow ing project by running a utoma ted tes tings

- linuxboot

- OpenBMC

Main principles• CI must be used from a

W eb API. Job Controller ta kes a s a n input a job description file w hich ca n s top the build a t a ny 4 predefined s ta ge.

• Output is a s ta tus a nd a rom file for a specific boa rd

• Job s ta tus ca n be queried through the API

job controller

builder

node brick va lida tion

ba s ic fea tures va lida tion

O/S s ta bility a nd s tress tes ts

W eb API

Active dev

Implementation1. Automa tic redeployment ma na ged through Ansible scripting2. Automa tic redunda ncy a nd sca la bility ba sed on ha rdw a re a va ila bility3. Slurm ba tch scheduler to ma na ge run queue, unique job ID, a nd jobs output4. KVM used to sa nd box builds into Ubuntu Xenia l VMs5. Ba s ic setup requires :

b. An a ns ible ma ster nodec. A Slurm controller noded. A Slurm ba tch node e . All of them sha ring the sa me subnet

Web API

GOALS1. Get a s a n input a g ithub

repository a ddress w ith a unique commit ID

2. Provide job control- La unch- Kill- Lis t

3 . Provide jobs s ta tus feedba ck- Build log file

INITIAL IMPLEMENTATION• W ritten in Go• Support

- Job la unch- Job s ta tus query- Job log

Job controller

GOALS1. Alloca te a nd ma na ge build nodes resources2. Preset build environment3. Store jobs s ta tus 4. Controlled us ing a hidden file w hich conta ins jobs description tha t w ill

override defa ult va lues

Job controller

INITIAL IMPLEMENTATION1. Alloca te job through Slurm ba tch scheduler2. Setup a virtua l ma chine (ba sed on Ubuntu Xenia l) w hen node is a lloca ted

w ith predefined cha ra cteris tics us ing KVM a nd virsh a. VM storage is seating in memory (about 40 GB)b. VM have access to the Internet and can run apt command to setup build

environment

3. Setup remote a ccess to the VM from the s lurm compute node4. Copy the re leva nt files into the VM a nd setup build environment5. Initia te job execution

BuilderGOAL• Build a fully functiona l linuxboot ima ge ba sed on job pa ra meters

INITIAL IMPLEMENTATION Ba sed on osresea rch/hea ds build environment

• Requires initia l boa rd ROM• Ca n build the linuxboot kernel (currently ba sed on 4.9 but ca n be cha nged)• Ca n a pply va rious pa tches to the kernel (BDS, AHCI etc …) ba sed on boa rd descriptor• Ca n build NERF (go ba sed) user environment (w ith full integra ted comma nd control) a s w ell

a s Hea ds (busybox/libc ba sed)• Ca n control/extra ct fina l DXE drivers integra ted w ithin linuxboot ROM• Genera te fina l linuxboot rom • Hea vy Ma kefile usa ge (looks like to be a “show stopper” to ma ny Z genera tion engineers)

Node brick validation

GOAL• Va lida te tha t a new ly built ima ge doesn’t brick a node (a ka tha t w e ca n ta lk

to the firmw a re through seria l a nd successfully execute ba s ic comma nd)

INITIAL IMPLEMENTATION• Ba sed on initia l qemu la unch of the ROM• Ba sed on rea l ha rdw a re setup w ith a ROM emula tor connected to the boa rd

Basic features validation

GOALS• Va lida te tha t a new ly built ima ge is properly detecting ha rdw a re• Va lida te tha t a new ly built ima ge is a ble to ins ta ll a n O/S a nd boot it through

- PXEboot or a ny other netw ork boot ca pa bility- loca l boot on AHCI a nd NVMe stora ge

Basic features validation

INITIAL IMPLEMENTATIONO/S ima ge ins ta ller built for linuxboot• Ubuntu Xenia l netins ta ller or loca l boot kernel fa ils to boot properly on

linuxboot (the kernel ha ngs)i. W e regenera te a full ISO ima ge ba sed a va lid origina l Ubuntu Xenia l ISO a nd

disa ble EFI support w ithin the ins ta ller kernel

ii. Tha t ima ge is a lso pre-configuring console output e ither to ttyS0 (loca l seria l) or ttyS4 (SoL) on W interfell ma chine

iii. Tha t new ima ge is a lso boota ble through a PXEboot process w hich is a utoma tica lly configured on the s lurm controller node.

O/S stability and stress testGOALS

Va lida te tha t a new ly built ima ge is a ble to:Run Linux a t full opera tiona l mode

i. Properly detect ha rdw a re beha vior (ACPI, CPU Frequencies ma na gement, Pow er Ma na gement, NUMA topology, VMX)

ii. Detect a nd ca n ma na ge w ithout error PCIe subsystems1. High speed netw ork a da pter (10 to 100 Gpbs)2. High speed s tora ge box (lightning)3. Low speed s tora ge box (knox)

iii. Can run various workload without system error and within an acceptable performance goal1. Run the Linpa ck/pysthone benchma rk a t speed2. Run Netw orking benchma rk a t speed3. Run bonnie++ a t speed4. Run Multiple VM implementa tions of the previous benchma rks

Using the CI as of today

Launching a build• Setup into a g it repo a .ci.yml file w hich conta ins 2 ba s ics entries

• Execute a protected curl reques t to http://<my_ci_server>:1234/v1/jobs getting a repo a nd bra nch to run from a s input pa ra meters

script -> Drive the execution w orkflow

a rtifa cts -> Expected output files

Using the CI as of today

Controlling executioncurl reques t to http://<my_ci_server>:1234/v1/jobs /<job_id>

Monitoring outputcurl reques t to http://<my_ci_server>:1234/v1/jobs /<job_id>/logs?ra w =true

Retrieving results - setting up complex inputThe build script sha ll ta ke ca re a bout pos ting re leva nt output results into a n end user repos itory or ftp build environment. Input like orig ina l ROM file sha ll be dow nloa ded by the build script into the re leva nt repos itory before build is kicked off.

What is next: PRC lab

• Live infra s tructure ba sed in Ca lifornia , Open to the community to tes t a nd va lida te firmw a re build

• Hosted by ITRenew• Dua l ra cks ba sed on OCP W interfell/Leopa rd servers

Call to Action

• Join ITRenew Open Ha rdw a re la b in s ilicon va lley (upcoming meetups)• Contribute to Open Source Firmw a re initia tive • Provide ha rdw a re for tes ting a nd build ea rly prototype• Conta ct us if you a re w illing to build a da pted des igns!• Dow nloa d des ign files on github!

• W a nt to commit code ? https ://g ithub.com/linuxboot/linuxboot-ci