Introduction to Integration CSITSanjib Mohapatra, Technical Lead, EricssonSumesh Kumar R, Technical Lead, EricssonSindhuri B, Senior Engineer, Ericsson

17/11/2016

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency. ”

- Bill Gates

Agenda• Introduction to Integration/test

• CI Infrastructure• Introduction to key terminologies (RF, Git, Gerrit, Jenkins)• Pre-requisites, Base Topology, Controller Installation, Getting started• CSIT directory structures • Important CSIT Libraries • Important CSIT variables

• How to start a test run and knowing dependency • Important Robot framework libraries• Basics of writing a test suite• Script Run time arguments • Test Coding and Submission Guidelines• Sample run of Clustering Test Suites

• How to upstream your script• Basic git commands for pushing scripts and raising gerrit review• How to cherry-pick and push an amend patch

• How to run your script in sandbox • How to setup jjb environment and create a job • How to run up-streamed scripts in sandbox

• Q & A

CI Infrastructure

Why Robot framework • The open source Robot Framework (RF) is a generic, keyword- and data-driven test

automation framework for acceptance test driven development (ATDD).• The RF is written in Python. Consequently, it runs on Python, Jython or IronPython.

The framework can thus be used with any OS that is able to run any of these interpreters, e.g. Windows, Linux or OS X.

• RF integrates well with Jenkins (formerly Hudson) which can start the tests and track results.

• Reporting and Logging capabilities of the Robot Framework are really great and are helping a lot in troubleshooting problems in the test implementation.

• Testing against a variety of interfaces (e.g. against REST and some UI) or against a multitude of product configurations/stacks/databases.

• Integration within a continuous delivery pipe line and with the test code repository.

What is Git, Gerrirt, Jenkins• Git is a version control system that is widely used for software developmentand

other version control tasks. It is a distributed revision control system with an emphasis on speed, data integrity, And support for distributed, non-linear workflows. As with most other distributed version control systems, and unlike most client–servers systems, every Git working directory is a full-fledged repository with complete history and full version-tracking capabilities, independent of network access or a central server.

• Gerrit is a free, web-based team code collaboration tool. Software developers in a team can review each other's modifications on their source code using a Web browser and approve or reject those changes. It integrates closely with Git, a distributed version control system.

• Jenkins is a Continuous Integration server. Continuous Integration is the practice of running your tests on a non-developer machine automatically every time someone pushes new code into the source repository.

This has the tremendous advantage of always knowing if all tests work and getting fast feedback. The fast feedback is important so you always know right after you broke the build (introduced changes that made either the compile/build cycle or the tests fail) what you did that failed and how to revert it.

Pre-requisites (software)• Python 2.7• Robot framework 3.0• robotframework-requests-0.4.5• requests-2.10.0• pystache-0.5.4• Clone latest integration/test master• Clone latest releng/builder master• jenkins-job-builder version 1.6.1• virtualenvwrapper 4.7.1.dev13• OVS 2.3.2

** The above software list and versions might change in due course of time**

Pre-requisites (VM)

VM , ODL, Cluster Configuration

VM OS Ubuntu 14.04 LTS

VCPU 10 VCPU

RAM 16G

ODL JVM Heap 8G

CPU Pinning for ODL process No

Cluster VMs networking All 3 VMs running in single host

No of Cluster nodes 3 VMs

All default ODL cluster settings used No changes for initial run

Topology

How to install 3Node cluster• Installing 3 node cluster in ODL way

Unzip the distribution in each node and go to following path

root@mininet-vm:/home/mininet/controller-Be/distribution-karaf-0.4.3-SNAPSHOT/bin# ls

client configure_cluster.sh instance karaf setenv shell start status stop

client.bat custom_shard_config.txt instance.bat karaf.bat setenv.bat shell.bat start.bat status.bat stop.bat

Execute in Node 1

root@mininet-vm:/home/mininet/controller-Be/distribution-karaf-0.4.3-SNAPSHOT/bin#

./configure_cluster.sh 1 10.183.181.51 10.183.181.52 10.183.181.53

Execute in Node 2

root@mininet-vm:/home/mininet/controller-Be/distribution-karaf-0.4.3-SNAPSHOT/bin#

./configure_cluster.sh 2 10.183.181.51 10.183.181.52 10.183.181.53

Execute in Node 3

root@mininet-vm:/home/mininet/controller-Be/distribution-karaf-0.4.3-SNAPSHOT/bin#

./configure_cluster.sh 3 10.183.181.51 10.183.181.52 10.183.181.53

• Install required features in all the controller nodes manually or by setting it in featuresBoot parameter of etc/org.apache.karaf.features.cfg and restarting the controller nodes.

Verify 3Node Cluster Ports

• Please check following port status before proceeding furtherroot@mininet-vm:~# cd /home/mininet/controller-Be/deploy/current/odl/bin

root@mininet-vm:/home/mininet/controller-Be/deploy/current/odl/bin# netstat -na | grep 2550

tcp6 0 0 10.183.181.51:2550 :::* LISTEN

tcp6 0 0 10.183.181.51:36101 10.183.181.52:2550 ESTABLISHED

tcp6 0 0 10.183.181.51:35895 10.183.181.53:2550 ESTABLISHED

tcp6 0 0 10.183.181.51:2550 10.183.181.53:37573 ESTABLISHED

tcp6 0 0 10.183.181.51:2550 10.183.181.52:58155 ESTABLISHED

root@mininet-vm:/home/mininet/controller-Be/deploy/current/odl/bin# netstat -na | grep 6633

tcp6 0 0 :::6633 :::* LISTEN

root@mininet-vm:/home/mininet/controller-Be/deploy/current/odl/bin#

Cloning Integration testroot@mininet-vm:/home/mininet/integration# git clone ssh://SanjibMohapatra@git.opendaylight.org:29418/integration/test

Cloning into 'test'...remote: Counting objects: 1711, doneremote: Finding sources: 100% (46/46)remote: Total 14028 (delta 4), reused 14013 (delta 4)Receiving objects: 100% (14028/14028), 2.46 MiB | 253 KiB/s, done.Resolving deltas: 100% (8310/8310), done.

root@mininet-vm:/home/mininet/integration# ls

distributions features feature-selector packaging pom.xml test vm

root@mininet-vm:/home/mininet/integration/test # ls

csit tools tox.ini

root@mininet-vm:/home/mininet/integration /test# cd tools/

root@mininet-vm:/home/mininet/integration /test /tools# ls

clustering fastbgp netconf_tools odl-mdsal-clustering-tests odl-ovsdb-performance-tests robot_check wcbench deployment mdsal_benchmarkodl-lispflowmapping-performance-tests odl-openstack-integration pcep_updater tools_vm wstools

Important directories inside CSITroot@ubuntu:/tmp/test/csit# lsconfigplans libraries scriptplans scripts suites testplans variables

root@ubuntu:/tmp/test/csit/libraries# ls Bulk* Karaf* Utils* Cluster* Mininet*BulkomaticKeywords.robot ClusterManagement.robot ClusterOvsdb.robot KarafKeywords.robot Utils.robot ClusterOpenFlow.robotMininetKeywords.robotroot@ubuntu:/tmp/test/csit/suites# lsaaa centinel integration mdsal netide openstack sfc tsdr vpnservicealto controller iotdm messaging4transport netvirt ovsdb snmp ttp vtnarmoury didm l2switch natapp nic packetcable sxp unimgrbgpcep distribution lacp nemo ocpplugin persistence test usccapwap groupbasedpolicy lispflowmapping netconf openflowplugin sdninterfaceapp topoprocessing usecpluginroot@ubuntu:/tmp/test/csit/testplans# ls openflowplugin-bulk*openflowplugin-bulk-matic-ds-daily-lithium-redesign.txt openflowplugin-bulk-matic-ds-daily.txt openflowplugin-bulkomatic-clustering-daily.txtroot@ubuntu:/tmp/test/csit/testplans# cat openflowplugin-bulkomatic-clustering-daily.txt# Place the suites in run order:integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/root@ubuntu:/tmp/test/csit/testplans#

*** scriptplans are executed before controller is installed and configplans are executed after contoller install but before it boots ***

CSIT Suites (Ex)• root@mininet-

vm:/home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic# ls

010__Cluster_HA_Data_Recovery_BulkFlow_Single_Switch.robot 020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot 030__Cluster_Reconcilliation_Multi_DPN.robot

• root@mininet-vm:/home/mininet/integration/test/csit/suites/openflowplugin/Clustering# ls

010__Cluster_HA_Owner_Failover.robot 020__Cluster_HA_Leader_Follower_Failover.robot 030__Cluster_HA_Data_Recovery_Leader_Follower_Failover.robot

CSIT Variables, Input files and Command to run Suites

• PATH to Set CSIT Variables

/home/mininet/integration/test/csit/variables# cat Variables.py

• PATH to Suite Specific Input files (ex: Json files, uri files)

root@mininet-vm:/home/mininet/integration/test/csit/variables# ls

alto bgpuser DIDM lispflowmapping ocpplugin ovsdb restconf topoprocessing Variables.pyc bgpclusteringcarpeople distribution nemo ofplugin packetcable sfc unimgr vpnservice bgpflowspec centinel gbpnetconf openflowplugin pcepuser tcpmd5user Variables.py xmls

• Commands to Run Suite via command line

root@mininet-vm:/home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic# pybot -L TRACE -v MININET_USER:mininet -v USER_HOME:/home/mininet -v CONTROLLER:10.183.181.51 -v CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v USER:root -v PASSWORD:rootroot -v WORKSPACE:/home/mininet -v BUNDLEFOLDER:/controller-Be/deploy/current/odl -v DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v MININET_PASSWORD:rootroot -v ODL_OF_PLUGIN:helium 030__Cluster_Reconcilliation_Multi_DPN.robot

http://robotframework.org/robotframework/#standard-libraries

Basics of writing a test suite

Git Commit Procedure and Wiki Info• 1. Sign In to Gerrit Code Review, Generate and Add the SSH key ( in https://git.opendaylight.org)

• 2. git clone ssh://SanjibMohapatra@git.opendaylight.org:29418/integration/test

• 3. got to "test" directory

• 4. git config --global user.name "Sanjib Mohapatra"

• 5 .git config --global user.email "sanjib.mohapatra@ericsson.com"

• 6. scp -p -P 29418 SanjibMohapatra@git.opendaylight.org:hooks/commit-msg .git/hooks/

• 7. chmod 755 .git/hooks/commit-msg

• 8. copy new files/changes - to respective directories and execute “git status”

• 9. git add ( for adding new files)

• 10. git commit -s -a --amend (for amend commit)

• 11. git commit -a -s (for new commit)

• 12. execute “git status”

• 13. git push ssh://SanjibMohapatra@git.opendaylight.org:29418/integration/test.git HEAD:refs/for/master

Refer Important wiki links

https://wiki.opendaylight.org/view/Category:Git_and_Gerrit

https://wiki.opendaylight.org/view/GettingStarted:Pulling,_Hacking,_and_Pushing_All_the_Code_from_the_CLI

https://wiki.opendaylight.org/view/GIT_Cheat_Sheet

git status• root@ubuntu:/home/sanjib/csit_repo/test# git status

# On branch master

# Your branch is ahead of 'origin/master' by 1 commit.

# Changes not staged for commit:

# (use "git add/rm <file>..." to update what will be committed)

# (use "git checkout -- <file>..." to discard changes in working directory)

# modified: csit/libraries/BulkomaticKeywords.robot

# deleted: csit/libraries/BulkomaticOpenFlow.robot

# modified: csit/suites/openflowplugin/Clustering_Bulkomatic/010__Cluster_HA_Data_Recovery_BulkFlow_Single_Switch.robot

# modified: csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot

# modified: csit/suites/openflowplugin/Clustering_Bulkomatic/030__Cluster_Reconcilliation_Multi_DPN.robot

# deleted: csit/variables/openflowplugin/sal_add_bulk_flow_small_config.json

# deleted: csit/variables/openflowplugin/sal_del_bulk_flow_small_config.json

# deleted: csit/variables/openflowplugin/sal_get_bulk_flow_small_config.json

# Untracked files:

# (use "git add <file>..." to include in what will be committed)

# csit/variables/openflowplugin/sal_add_bulk_flow_config.json

# csit/variables/openflowplugin/sal_del_bulk_flow_config.json

# csit/variables/openflowplugin/sal_get_bulk_flow_config.json

no changes added to commit (use "git add" and/or "git commit -a")

git add• root@ubuntu:/home/sanjib/csit_repo/test# git add

csit/variables/openflowplugin/sal_add_bulk_flow_config.json• root@ubuntu:/home/sanjib/csit_repo/test# git add

csit/variables/openflowplugin/sal_del_bulk_flow_config.json• root@ubuntu:/home/sanjib/csit_repo/test# git add

csit/variables/openflowplugin/sal_get_bulk_flow_config.json• root@ubuntu:/home/sanjib/csit_repo/test# git add csit/libraries/BulkomaticKeywords.robot• root@ubuntu:/home/sanjib/csit_repo/test# git add

csit/suites/openflowplugin/Clustering_Bulkomatic/010__Cluster_HA_Data_Recovery_BulkFlow_Single_Switch.robot

• root@ubuntu:/home/sanjib/csit_repo/test# git add csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot

• root@ubuntu:/home/sanjib/csit_repo/test# git add csit/suites/openflowplugin/Clustering_Bulkomatic/030__Cluster_Reconcilliation_Multi_DPN.robot

git commit• root@ubuntu:/home/sanjib/csit_repo/test# git commit -s -a --amend[master e84e914] Reconcilliation in a 3 node cluster multi DPN environment7 files changed, 241 insertions(+), 69 deletions(-)create mode 100644

csit/suites/openflowplugin/Clustering_Bulkomatic/030__Cluster_Reconcilliation_Multi_DPN.robotrename csit/variables/openflowplugin/{sal_add_bulk_flow_small_config.json =>

sal_add_bulk_flow_config.json} (90%)rename csit/variables/openflowplugin/{sal_del_bulk_flow_small_config.json =>

sal_del_bulk_flow_config.json} (90%)rename csit/variables/openflowplugin/{sal_get_bulk_flow_small_config.json =>

sal_get_bulk_flow_config.json} (100%)

*** After successful execution of git commit, git log file will open, code committer needs to put proper commit message and exit out of log file. Use “git log” command to see the commit message again***

git push• root@ubuntu:/home/sanjib/csit_repo/test# git push ssh://SanjibMohapatra@git.opendaylight.org:29418/integration/test.git

HEAD:refs/for/masterCounting objects: 25, done.Delta compression using up to 8 threads.Compressing objects: 100% (15/15), done.Writing objects: 100% (15/15), 3.83 KiB, done.Total 15 (delta 10), reused 0 (delta 0)remote: Resolving deltas: 100% (10/10)remote: Processing changes: updated: 1, refs: 1, doneremote:remote: Updated Changes:remote: https://git.opendaylight.org/gerrit/40365 Reconcilliation in a 3 node cluster multi DPN environmentremote:To ssh://SanjibMohapatra@git.opendaylight.org:29418/integration/test.git* [new branch] HEAD -> refs/for/masterroot@ubuntu:/home/sanjib/csit_repo/test#

git cherry-pick• root@ubuntu:/home/sanjib/csit_repo/test# git fetch https://git.opendaylight.org/gerrit/integration/test

refs/changes/65/40365/4 && git cherry-pick FETCH_HEAD

remote: Counting objects: 1718, done

remote: Finding sources: 100% (9/9)

remote: Total 9 (delta 0), reused 2 (delta 0)

Unpacking objects: 100% (9/9), done.

From https://git.opendaylight.org/gerrit/integration/test

* branch refs/changes/65/40365/4 -> FETCH_HEAD

[master 62b79bc] Reconcilliation in a 3 node cluster multi DPN environment

2 files changed, 182 insertions(+)

create mode 100644 csit/libraries/BulkomaticOpenFlow.robot

create mode 100644 csit/suites/openflowplugin/Clustering_Bulkomatic/030__Cluster_Reconcilliation_Multi_DPN.robot

Integration/test Coding and GerritSubmission Guidelines

• For both REST and SSH actions, Use instance indexes as preferred arguments for cluster KWs.

• So far controllers REST sessions are opened at the beginning and maintained through the test, SSH sessions are opened on when required. Hence, avoid creating repeated session instance using IP address, instead make use of controller sessions that are opened in the (Suite Setup:: Create Controller Sessions) suite start-up.

• Invoke (Suite Teardown ::Delete All Sessions) during suite start-up.

• Repeated work in main test suites to be put inside libraries.

• Use Tidy Tool for cleaning up and changing format of Robot Framework test data files before committing script in Integration/test (ex: python -m robot.tidy --inplace tests.robot).

• Use proper commit message, Check out below link on writing good commit messages http://chris.beams.io/posts/git-commit/

• For comprehensive integration test coding guideline follow below link https://wiki.opendaylight.org/view/Integration/Test/Test_Code_Guidelines

Creating sandbox environment• If you don't have RelEng/Builder's repo, clone it “git clone https://git.opendaylight.org/gerrit/p/releng/builder.git “.

• If existing RelEng/Builder's repo is there then rebase to latest releng/builder master and edit your jenkins.ini according to

[http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#jenkins-sandbox].

• install Virtualenvwrapper to manage JJB and its Python dependencies (http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#installing-jenkins-job-builder )

• # Make a copy of the example JJB config file (in the builder/ directory)

$ cp jenkins.ini.example jenkins.ini

# Edit jenkins.ini with your username, API token and ODL's sandbox URL

root@ubuntu:/home/sanjib/builder# cat jenkins.ini

[job_builder]

ignore_cache=True

keep_descriptions=False

include_path=.:scripts:~/git/

recursive=True

[jenkins]

user=SanjibMohapatra

password=0c984e63309dc9336728ff064c1b277

url=https://jenkins.opendaylight.org/sandbox

query_plugins_info=False

##### This is deprecated, use job_builder section instead

#ignore_cache=True

Jenkins sandbox wiki

• Make sure your jenkins-job-builder version is at least 1.6.0

• http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#jenkins-sandbox

• If your project requires access to the sandbox please open an OpenDaylight Helpdesk ticket (<helpdesk@opendaylight.org>) and provide your ODL ID.

Create .yaml file• root@ubuntu:/home/sanjib/builder/jjb/openflowplugin# cat openflowplugin-csit-bulkomatic-clustering-testing.yaml

- project:

name: openflowplugin-csit-bulkomatic-clustering-testing

jobs:

- '{project}-csit-3node-{functionality}-{install}-{stream}'

# The project name

project: 'openflowplugin'

# The functionality under test

functionality: 'bulkomatic-clustering-testing'

# Project branches

stream:

- boron:

branch: 'master'

jre: 'openjdk8'

install:

- only:

scope: 'only'

# Features to install

install-features: 'odl-restconf-all,odl-clustering-test-app,odl-netconf-connector-ssh,odl-openflowplugin-all-li,odl-openflowplugin-app-bulk-o-matic-li'

# Robot custom options

robot-options: '-v ODL_OF_PLUGIN:lithium'

# Trigger jobs (upstream dependencies)

trigger-jobs: 'yangtools-distribution-{stream},controller-distribution-{stream},openflowjava-distribution-{stream},openflowplugin-distribution-{stream}'

root@ubuntu:/home/sanjib/builder/jjb/openflowplugin#

Creating jjb (test)root@ubuntu:/home/sanjib/builder# source /usr/local/bin/virtualenvwrapper.sh

root@ubuntu:/home/sanjib/builder# mkvirtualenv jjb

New python executable in /root/.virtualenvs/jjb/bin/python

Installing setuptools, pip, wheel...done.

(jjb) root@ubuntu:/home/sanjib/builder# workon jjb

(jjb) root@ubuntu:/home/sanjib/builder# jenkins-jobs --conf jenkins.ini test jjb/ openflowplugin-csit-3node-bulkomatic-clustering-testing-only-boron

INFO:jenkins_jobs.local_yaml:Including file 'include-raw-jjb-update.sh' from path '/home/sanjib/builder/jjb'

INFO:jenkins_jobs.local_yaml:Including file 'include-raw-check-poms.sh' from path '/home/sanjib/builder/jjb‘

..........

INFO:jenkins_jobs.builder:Number of jobs generated: 1

INFO:jenkins_jobs.builder:Job name: openflowplugin-csit-3node-bulkomatic-clustering-testing-only-boron

<?xml version="1.0" encoding="utf-8"?>

<project>

</com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper>

</buildWrappers>

</project>

(jjb) root@ubuntu:/home/sanjib/builder#

*** If the job you’d like to test is a template with variables in its name, it must be manually expanded before use. For example, the commonly used template '{project}-csit-3node-{functionality}-{install}-{stream}‘ to openflowplugin-csit-3node-bulkomatic-clustering-testing-only-boron ***

Creating jjb (update) • (jjb) root@ubuntu:/home/sanjib/builder# jenkins-jobs --conf jenkins.ini update jjb/ openflowplugin-csit-3node-bulkomatic-clustering-testing-only-boron

INFO:root:Updating jobs in ['/home/sanjib/builder/jjb', '/home/sanjib/builder/jjb/packetcable', '/home/sanjib/builder/jjb/netconf', '/home/sanjib/builder/jjb/dlux',

'/home/sanjib/builder/jjb/openflowplugin', '/home/sanjib/builder/jjb/snmp4sdn', '/home/sanjib/builder/jjb/lacp', '/home/sanjib/builder/jjb/nic', '/home/sanjib/builder/jjb/alto', '/home/sanjib/builder/jjb/usc', '/home/sanjib/builder/jjb/toolkit',

…

…

…

INFO:jenkins_jobs.local_yaml:Including file 'include-raw-setup-docker.sh' from path '/home/sanjib/builder/jjb/ovsdb'

INFO:jenkins_jobs.local_yaml:Including file 'include-raw-cleanup-docker.sh' from path '/home/sanjib/builder/jjb/ovsdb'

INFO:jenkins_jobs.local_yaml:Including file 'include-raw-setup-docker.sh' from path '/home/sanjib/builder/jjb/ovsdb'

INFO:jenkins_jobs.local_yaml:Including file 'include-raw-cleanup-docker.sh' from path '/home/sanjib/builder/jjb/ovsdb'

WARNING:root:logrotate is deprecated on jenkins>=1.637, use the property build-discarder on newer jenkins instead

INFO:jenkins_jobs.builder:Number of jobs generated: 1

INFO:jenkins_jobs.builder:Creating jenkins job openflowplugin-csit-3node-bulkomatic-clustering-testing-only-boron

INFO:root:Number of jobs updated: 1

(jjb) root@ubuntu:/home/sanjib/builder#

******* The successful yaml file needs to be uploaded to master Releng/builder, thus ensuring regular job run in Jenkins *******

https://jenkins.opendaylight.org/sandbox/login

Locate the jjb and click on it

Inside the job

Configure

Build with Parameter

*** Ensure proper Test plan and patchrefspec is given ***

Rebuild Last with Parameter

*** Ensure proper Test plan and patchrefspec is given ***

Inside Build

Inside Console Log

Browse Test results and share it

Conclusion

• Continuous system integration is the order of the day.• Get familiar with Git, Gerrit, Jenkins it will help immensely to get close with

ODL way of workings.• It is equally important to have hands-on with "sandbox”, any commit with

sandbox results adds credibility to what you have written and has greater chance of getting submitted easily.

Q & A