How to use OPNFV Testing tools to test/validate an ... · Intel Ericsson Huawei OpenDayLight ONOS...

How to use OPNFV Testing tools to test/validate an Openstack

provided by a vendor

Morgan Richomme

NFV architect at Orange Labs

OPNFV ambassador

ONAP & OPNFV Functest committer

Speakers

Valentin Boucher

Software engineer at Kontron Canada

OPNFV Functest committer: https://wiki.opnfv.org/display/functest

Agenda

● OPNFV testing overview • OPNFV Testing chain

• OPNFV Testing tools

● Reusing those tools• Kontron

• Orange

OPNFV Testing OverviewTesting chain and tools …..

OPNFV CI complexity

Addons / Features

SDN Controller

Installer / Deployer

Lab/POD

SFC BGPVPN DPDK FDio

APEX Compass Daisy Joid

Linux Foundation Intel Ericsson Huawei

OpenDayLight ONOS OpenContrail

Fuel XCI

OPNFV Testing chain

ScenarioHA/noHA, SDN,

DPDK etc..

DeployerAPEX, Compass,

Daisy, Fuel and JOID

PodLF, Huawei, Intel etc ..

VersionMaster, Fraser

DeployInstaller project

ReportTest api, reporting

TestFunctest,

Yardstick etc

Test result

OPNFV Testing ecosystem

Functional testing -> Functest

Functest● Answer the question: “Does my infrastructure simply work as expected?”

● Provide tests from basic connectivity (<1s) to VNF testing (>1h)

● Leverage Upstream test suites from the integrated components (OpenStack/Rally & Tempest, Kubernetes/Default test suites, OpenDaylight, …)

● Allow “A la carte” test combination through rich, light and modular Alpine based framework, adapted to CI/CD chains

● PTL: C.Ollivier, Orange

Functest testcases for OpenStackC

ompl

exity

Gating

Official R

elease validation

Target validation

Full stack

Performance testing -> Yardstick & cie

Yardstick● Answer the question: “Qualify my NFVI and give me valuable figures?”

● Qualify Compute, Network and Storage domains through elementary tests

● Leverage OPNFv community to build baseline and define acceptable thresholds for Telco Cloud solution

● Allow lab owner to compare their Telco Cloud KPI with the other OPNFV labs

● In line with ETSI NFV

● PTL: R. Brattain, Intel

Yardstick

More than 50 available tests covering different domains

(compute, network, storage, HA, IPv6, KVM, ..)

Src: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html

Yardstick

Src: Alassane Samba (Orange) https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results

TC_012 post processing

Yardstick testing capacity

• Generic test cases

• Compute

• Storage

• Network

• HA test cases

• Service HA

• Node HA

• Function test cases

• SRIOV

• Dpdk

• Hugepages

Certification -> Dovetail

Dovetail● Answer to question: “Do these products conform to the interfaces,

behaviors and functionalities defined by OPNFV?”

● Main objective: reduce VNF onboarding cost

● Do not implement test cases in their own code base but reuse the test cases from the OPNFV test projects (Functest, Yardstick etc..)

● Allow vendors to use the OPNFV verified logo

● PTL: Georg Kunz, Ericsson

Dovetail – Test scopeWhole OPNFV testsuite

OPNFV Verified 2018.01

Mandatory test cases● OpenStack API tests (205 tests)● Layer 2 packet forwarding (2 tests)● OpenStack control service HA (8 tests)

Optional test cases● IPv6 tenant networks (25 tests)● BGPVPNs (4 tests)● Fundamental VIM capabilities (30 tests)

Future test case candidates● Neutron Trunk ports ● BGPVPN Tempest tests● SDN Controller HA● VNF lifecycle management● ….

Future ..

Certification workflow

1. Submission of participation form

2. Testing

3. Submission of results

4. Notification of reviewers

5. Review of test results

6. Grant of use of program marks

Reuse those tools

Kontron communications unit (Canada): part ODM, part integrator

SYMKLOUD hw and solution portfolio: integrate open source with high density converged platforms + cloud servers + white box switchesUsed for: NFVi at Edge + Core, SDN, DC clouds for the telco and video/broadcast, and cloud providers We make it easy: removed complexities to integrate software (CI/CD open source or not) on Symkloud hw; plus can ‘mix-n-match’ composable modules (GPU, FPGA, Crypto, Storage).

Ramp up services faster and with lower risk: easy to Configure, Deploy and Manage services on our platforms (automated switch configuration and deployment, integrate addons,etc.)

Context

Problematics

How to test/validate that our deployment of Openstack or Kubernetes on SymKloud is fine ?

● Different SymKloud Platforms○ Different type of switches (40Gb, 100Gb/SDN-ready etc..)○ Different CPU (12, 16 .. cores)○ Servers with/without GPU or FPGA etc…

● Differents software type and version(OpenStack Newton/Queens, Kubernetes 1.9/1.10 etc..)

● Differents add-ons (6wind network acceleration for OpenStack, OpenStack Ironic support etc..)

More features can be added in the future (New deployer, SDN controller, New hardware)

Current CI

ScenarioHA/noHA, ironic,

6wind etc..

Cloud softwareOpenStack, Kubernetes

Hardware

VersionArnold, Britney, Develop etc..

DeployMAAS/Juju

ReportTestrail

TestHomemade

test suite using Behave

Experimental CI

ScenarioHA/noHA, ironic,

6wind etc..

Cloud softwareOpenStack, Kubernetes

Hardware

VersionArnold, Britney, Develop etc..

DeployMAAS/Juju

ReportTest api, reportingTestrail

Test

Test result

TestHomemade

test suiteFunctest,Yardstick

KontronOPNFV tools

Reporting

Our goals

Why reuse those tools in addition of our internal QA test ?• Our QA test suite has a more diverse test coverage than the OPNFV

tools (not only focusing on the NFVI+VIM test but also installer, add-ons, configuration and hardware test)

• The OPNFV test suite has more tests related to NFV so it completes our test coverage

• The goal is to be better and close to the customer target usage• We ensure that next Dovetail verification will work if all opnfv tests pass

on our platform.

The Orange CI/CD view (Preproduction)

CI/CD viewn ONAP OpenLABBare Metal O/S management

Infrastructure Installation

ONAP Installation

Openstack BIFROST / OPNFV XCI

Kolla Ansible

ONAP Heat

Kubespray

Functest Xtesting(Healthchecks/moke/vnf //

smoke/conformance)

ONAP OOM

Functest Xtesting/onap-robot

Run daily

Run daily

Test DB

Test

AP

I

ONAP Heat (Community lab)

OPNFV Orange Integration Center

Installers 5: Apex, Compass, Daisy, Fuel, Joid Preproduction: 1 (RedHat Director)Orange ONAP OpenLab: 1 Kolla Ansible

Scenarios Mores than 30 Preproduction: 1 OpenStack + ContrailOrange ONAP OpenLab: 2 OpenStack / Kubernetes + ONAP

Tests Tests (automation): relying on several test projects:

● Functest* for functional testing● Yardstick* for performance testing● Qtip for bench HW● Storeperf for ceph● Vsperf for Vswitch qualification● Bottleneck to optimize Yardstick runs

* used for release validation

Preproduction: Functest + YardstickOrange ONAP OpenLab: Functest + ONAP Robot test suites

Reuse of OPNFV tooling for ONAP

Reuse of Functest Xtesting framework to onboard ONAP robot tests• Only 3 files needed (Robot natively supported)• Docker size from 1 Go to 100Mo (Alpines)• Inherit Test reporting capabilities• Consistent view to manage Infra and orchestration testing (reuse

DB, Test API)

Reuse of OPNFV tooling for ONAP

● Test framework● Test API/Test DB● CI/CD methodology

Conclusions

• OPNFV testing tool is useful, evolutive, promoting Upstream First (Openstack/ODL testing projects), it can easily complete existing vendor suites with a good focus on NFV

• Automation is key: Xtesting framework is very well adapted to this context (light (Alpines), simple, modular)

• Tooling is easily reusable out of OPNFV context (ONAP, Openstack/Kubernetes - non OPNFV - solutions) including frameworks, DB, test API

• OPNFV release/Dovetail criteria should be stronger as test frameworks offer more than what is used as release criteria: infrastructure trustability is key for Telco Cloud/NFV adoption

Thank You!

How does Functest work?

Yardstick Architecture

• User Interface

• Main Framework

• Context

• Scenario

• Runner

• Dispatcher

• Plugin

• Container Manager

• Services

OPNFV Test result API

http://artifacts.opnfv.org/releng/docs/testapi.html

case_name: "cloudify_ims",pod_name: "huawei-pod2",version: "fraser",installer: "compass",scenario: "os-odl_l3-nofeature-ha",start_date: "2018-04-18 04:58:10",stop_date: "2018-04-18 05:38:50",criteria: “PASS” ,build_tag: "jenkins-functest-baremetal-22",details: { }

name: "cloudify",version: "4.0",status: "PASS",duration: 349

name: "clearwater"descriptor_version: "122",version: "107",status: "PASS",duration: 1330

name: "clearwater-live-test”,version: "1.0",status: "PASS",duration: 273,result: {failures: 0, skipped: 30, passed: 46}

orchestrator

VNF

VNF Test

Result of each run in OPNFV test API

OPNFV Reporting

http://testresults.opnfv.org/reporting/

Trend, score, iteration by installer, version, scenario

OPNFV Reporting

http://testresults.opnfv.org/reporting/

Last test result by installer, version, scenario