Linaro Connect,Hong KongMarch 2013

July 2013

LCE13LNG Testing and Benchmarking

connect.linaro.org

Agenda

● Discussion on how to enhance LAVA framework for networking applications.

● Discussion on LNG testing strategy and goals.

https://lce-13.zerista.com/event/member/79674

http://pad.linaro.org/p/LCE13_LNG-Benchmarking

connect.linaro.org

● Why: Make sure we don't have regression in functionality and performance. Regression could be introduced by upstream or by us.

● What: Baselines and regression testing.

● How: LAVA.

For example: compare LNG kernel with and without RT patchset.

LNG Testing/Benchmarking

connect.linaro.org

LNG Focus Areas

● Big Endian

● Foundation○ LNG Kernel (LSK + Hugepage + NOHZ_FULL + RT...)

● Data Plane○ RTE + SOC APIs

What are the Key Performance Indicators?

connect.linaro.org

LNG Test Cases

LNG Tests/Benchmarks Tracking Sheet

● Functional Testing○ Additional patches carried in LNG kernel trees. Some have wider impact (e.g.

RT), some may be backported features (e.g. hugepage, nohz_full).○ Need coverage for additional features, as well as kernel in general post

integration.

● Performance Benchmarking○ Some features improve certain performance characteristics (e.g. hugepage,

nohz_full).○ Need to characterize those and make sure they continue to work well across

releases.

connect.linaro.org

LNG Test Cases

Measure:● Throughput● Number of dropped packets● Latency● CPU utilization● Power

Other Variables:● Number of cores● Core frequency● Number of ports

Test Variations:● Accelerated using Dataplane

APIs.● Inside KVM guest.

○ Guest-network.○ Guest-guest.

● Kernel with different patchsets (nohz_full, rt, etc.).

Example: network packet forwarding.

Compare against x86.

connect.linaro.org

LNG Test Bench (Network Traffic)

Tests with network traffic, especially for data plane applications.

DUT: member platforms, multi-10G, multi-1G ports.

Using x86 machines as traffic generator. Could also be DUT for comparative benchmarking.

connect.linaro.org

LNG Test Bench (Network Traffic)

Some ports directly connected between Traffic Generator and DUT.

Some ports connected through switch.

Can even repurpose some DUT as Traffic Generator.

connect.linaro.org

LNG Test Bench (Network Traffic)● Resource Management

○ Isolated data path between traffic generator and DUT.

○ Automated / programmatic way to access traffic generator, switch, etc.

○ Efficient resource scheduling.

● Test & Resource Coordination○ Start app on DUT, start

traffic, stop traffic, stop app on DUT.

● Test Report & Database○ Performance trends,

passing criteria, statistics○ Visualization, alerts.

connect.linaro.org

LNG Test Bench (w/o Network Traffic)

Tests without network traffic, primarily for kernel regression testing.

● Resource Management○ Automated / programmatic

way to access traffic generator, switch, etc.

○ Efficient resource scheduling.

● Test Report & Database○ Performance trends,

passing criteria, statistics○ Visualization, alerts.

connect.linaro.org

● What are the target applications? Example code?● Recommendations of how we should categorize

benchmarks?○ By verticals? e.g. wireless, security?○ L1-L7?

● Any additional hardware platforms to consider?● Wrt comparative benchmarking, which platforms do we care

about?● How much do we care about industry benchmarks such as

SPEC, EEMBC?● How do we characterize success of our data plane APIs?● In case of data plane, there are different solutions: bare

metal, Linux userspace RTE, etc. How do we showcase LNG solution?

●

Questions

Linaro Connect,Hong KongMarch 2013

Questions?

More about Linaro: http://www.linaro.org/about/More about Linaro engineering: http://www.linaro.org/engineering/

How to join: http://www.linaro.org/about/how-to-joinLinaro members: www.linaro.org/members