From System Modeling to Automated System Testing

From System Modeling to Automated System Testing

An overview of the CITK Toolchain

BOSCH — February 12th 2015. Stuttgart, Germany | Florian Lier [[email protected]]

Systems in Robotics | Current State

Research on autonomous robots achieved considerable progress over the recent years.

Reported results are typically validated through experimental evaluation or demonstrated live at robotics competitions. [ DARPA Challenge, RoboCup, …]

Amongst other reasons, this is due to: „off-the-shelf“ robots [ iCub, NAO, … ] and vivid open source communities providing reusable building blocks, e.g. ROS, Orocos, OPRoS, Yarp …

Publicly available data sets are used to improve benchmarking procedures, i.e., Rawseeds Project.

3

Systems in Robotics | Software Foundation

4

! Most current robotics systems are realized by implementing a component-based architecture

These components do not necessarily share the same ecosystem, build, deploy and testing environment or integration model …

AR Toolkit

Systems in Robotics | Software Foundation

Systems in Robotics | Problem Statement

System artifacts are often distributed over different locations, thus already the discovery, identification and aggregation of all required artifacts is difficult.

Which specific versions (master or v0.1) of software components were in use for a particular system or ideally system version?

How can automated integration testing, evaluation and preservation of results of these heterogeneous components be achieved?

7

Systems in Robotics | Aspect Overview

8

Before you can test

Systems in Experimental Robotics | Goal

To tackle these issues we introduced an approach based on an integrated software toolchain for system developers (testers) and experiment designers.

9

Systems in Experimental Robotics | System Model

10

Test!

n SCM’s

dev dev

iCub-nightly

auto-gen

auto deploy

/somewhere/$USER/

Recipes: Java, C/C++, Python, Experiments System Composite

Run Software & Experiment in Lab

direct use

..or in office/home

Local & Global CI

11

sync

Web-based catalog (Systems and related artifacts)

Systems in Robotics | Tool Chain

n SCM’s

dev dev

iCub-nightly

auto-gen

auto deploy

/somewhere/

Recipes: Java, C/C++, Python System Composite

Lab

direct use

Local & Global CI

12

Systems in Robotics | Technical Realization Recipes

..or in office/home

sync

Web-based catalog (Systems and related artifacts)

13

Systems in Robotics | Catalog

14

Systems in Robotics | Catalog

Local & Global CI

15

Systems in Robotics | Bootstrapping

download pre-packaged CI Server

Local & Global CI

16

Systems in Robotics | Bootstrapping

extract & simply start it

17

Systems in Robotics | Jenkins

find ready to use local (or global) CI Server

18

clone distribution/recipe repository

Systems in Robotics | Job Generator

19

invoke job-configurator with desired distribution including tests


20


Systems in Robotics | Testing

21

!

What are these “tests” actually?

Systems in Robotics | Test Framework Goals

22

Not a “tailored” solution for a specific system or even language (component-based architecture may even prevent this in the first place)

Minimize the effort, let the machine do the work (CI) and enable developers to design well-structured functional tests

Explicit environment setup and configuration, transparent orchestration, repeatable and comparable tests

Serialize building and testing automatically

Enable and verify system start-up and tear-down

23

SM

$ENVIONMENT_VARS_DEF

$COMPONENT_A(CHECK_CRITERIA)$COMPONENT_B(CHECK_CRITERIA)$COMPONENT_N(CHECK_CRITERIA)

DM

EXEC

RUN

CLEAN

EVAL

CLEAN / REPORT

exec(COMPONENT_A): if "OK" Bexec(COMPONENT_B): if "OK" Cexec(COMPONENT_C): if "OK" D

exec(COMPONENT_Y): if "OK" Z

inject()

exec(COMPONENT_Z): if "OK" E

EXIT

ERROR

export ROS_ROOT=/opt/ros/indigo export DISPLAY=:0

rosbag record -j /some/topicjava -jar slow_in_java.jarsh overly_complicated.sh

must start after 5 secondsmonitor PID continuouslymust not print “segfaulting()”must print “conn. established”lockfile.file must be present

Component Success Criteria

24

Systems in Robotics | Test State Machine Concept

SM



DM

EXEC

RUN

CLEAN

EVAL

CLEAN / REPORT



inject()


EXIT

ERROR

stop processes in RUN.invert()

e.g., plot.sh < $FSMT_RUN/logs/*e.g., read_and_clean.logs

stop processes in EVAL.invert()

25

SM



DM

EXEC

RUN

CLEAN

EVAL

CLEAN / REPORT



inject()


EXIT

ERROR

Close all component loggersReport exit status sys.exit(n)Create zip archiveCreate xUnit output

Systems in Robotics | Test State Machine Concept

26

Systems in Robotics | Test State Machine Implementation

Framework called FSMT (Finite State Machine Based Testing)*

Set of tools written in Python

Implementation of executable “test” state machines including all features as defined in the concept

Simple configuration using *.ini files

* http://opensource.cit-ec.de/projects/fsmt/

27


28


29

Systems Robotics | FSMT Invocation

fsmt_iniparser my_test.ini -o my_test.scxml

30

Systems Robotics | DEMO

DEMOhttps://vimeo.com/112005754

31

Systems Robotics | Provided Benefits

Not a “tailored” solution for a specific system or even language (component-based architecture may even prevent this in the first place)

Minimize the effort, let the machine do the work (CI) and enable developers to design well-structured functional tests

Explicit environment setup and configuration, transparent orchestration, repeatable and comparable tests

Serialize building and testing automatically

Enable and verify system start-up and tear-down

32

Systems Robotics | Thank you!

Links

Web Catalog: https://toolkit.cit-ec.de Distribution Project: https://opensource.cit-ec.de/citk Video Material: http://vimeo.com/groups/citkhttp://opensource.cit-ec.de/projects/fsmt/

Affiliation

Florian Lier is with Cognitive Interaction Technology — Center of Excellence, Bielefeld University Supervisor: Dr.-Ing. habil Sven Wachsmuth

33

Systems Robotics | Literature

Publications (2014)

Modeling Software Systems in Experimental Robotics for Improved Reproducibility -- A Case Study with the iCub Humanoid Robot http://pub.uni-bielefeld.de/publication/2705677

The Cognitive Interaction Toolkit – Improving Reproducibility of Robotic Systems Experiments http://pub.uni-bielefeld.de/publication/2702526

Towards Automated Execution and Evaluation of Simulated Prototype HRI Experimentshttp://pub.uni-bielefeld.de/publication/2645922

Simulation and HRI Recent Perspectives with the MORSE Simulatorhttp://pub.uni-bielefeld.de/publication/2685211

34

https://vimeo.com/112005754

Systems in Robotics | Static Demo