Embed Size (px)
Citation preview
Axel Rennoch, Sascha Hackel, Alexander Kaiser
TestCon 2019
04. April 2019, Moscow
INTERNET OF THINGS QUALITY ASSURANCE
2
BERLIN CENTER FOR DIGITAL TRANSFORMATION
3
• IoT Testing
− Challenges and scope
− IoT test language: TTCN-3
• Project IoT-T
− Eclipse IoT-Testware
− Standardization & Certification
• Summary and outlook
AGENDA
4
• Mirai botnet, October 2016:
− botnet using insecure configured IoT-devices (~100.000)
− attack causes blackout and disruption
(e.g. Amazon, Netflix, Twitter, Github)
• Wannacry, May 2017
− ransomeware affecting the whole world (e.g. hospitals in the U.K.)
• KRACK: Key Reinstallation Attack, October 2017
− Replay attack on Wi-Fi Protected Access protocol
• Spectre and Meltdown, January 2018
− Spectre: vulnerability that allows observable side effects from mispredicted
speculative executions
− Meltdown: hardware vulnerability that allows to read all memory
MOTIVATION FOR QUALITY
5
TRENDS IN IOT
6
IOT ARCHITECTURE
The Three Software Stacks Required for IoT Architectures, Eclipse IoT Working Group, September 2016
telemetry
commands
telemetry
commands
7
Connectivity options Throughput
Latency
Power efficiency
Packet size
Wide portfolio of competences required
Devices (sensors, HW, embedded SW)
Platforms (Cloud, platform domain knowledge)
Applications (SW, dashboard, business logic)
TECHNICAL SCOPE
IoT platforms 360+ worldwide
IoT protocols Rich selection
IP-based
non-IP based
Cellularnetw.
Cellular: 4G, NB-IoT, Cat-M1,EC-GSM…[5G];non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, … connectivity
MQTT MQTT-SN CoAPAMQP3OPCUA
XMPPHTTP1 WebSocket21|2|3
applicationIoT application logic
TCP UDPSMS
DTLSDTLS
IPv4/IPv6(6loWPAN)
transportTLS/SSL
IoT services layer services…
8
• IoT devices,
Mikrocontroller (MCU),
Gateways (Bosch XDK, IoT starterkits)
• IoT platforms
RIOT, relayr, Thread, mbed…
service layer (oneM2M, FiWare)
• IoT protocols
Constrained Application Protocol (CoAP)
MQ Telemetry Transport (MQTT)
IoT challenges: complexity, asynchronism, resource constraints, long operation phase
STARTING: TEST OBJECTS
LPWAN
LoRa
oneM2M
9
After the acceptance and system tests there will be
a long operation phase => new test phase „operation“
LONG OPERATION LIFETIME
10
INTEGRATION OF SEVERAL TESTING APPROACHES
IoTTesting
Software Testing
System Testing
Security Testing
Test Automation
Protocol Testing
11
• Less resources needed (time and money)
• Avoid human mistakes due to manually testing
• During test development and execution
• Speed-up of regression tests and product time-to-market
TEST AUTOMATION
12
MULTIPLE TEST CONFIGURATION (SAMPLES)
13
• Toolset (selection of available means)
Protocol tester/monitor (Eclipse Titan, Wireshark)
Test devices (RFID kit, Bluetooth test device)
GUI tester (Selenium, SikuliX, Chrome headless)
Web services tester (soapUI)
…
• Public Testsuites (in development)
Application of a standardized notation
Abstract and platform-independent
TESTWARE
14
IOT TEST LANGUAGE
15
• TTCN-3 is the Testing and Test Control Notation
• Internationally standardized testing language for formally defining test
scenarios.
• Designed purely for testing
CHALLENGE TEST AUTOMATION
testcase Hello_Bob () {
p.send(“How do you do?“);
alt {
[]p.receive(“Fine!“);
{setverdict( pass )};
[else]
{setverdict( inconc )} //Bob asleep!
}}
16
• One test technology for different tests
Distributed, platform-independent testing
Integrated graphical test development, documentation and analysis
Adaptable, open test environment
• Areas of Testing
Conformance and functional testing
Interoperability and integration testing
Real-time, performance, load and stress testing
Security testing
Regression testing
• Used for system and product qualification and certification
DESIGN PRINCIPLES OF TTCN-3
17
TTCN-3 Test Case
TTCN-3 IS DESIGNED FOR DYNAMIC TESTING
Port.send(Stimulus) Port.receive(Response)
System Under Test
Port
• Assignment of a
Test Verdict
18
IMPLEMENTATION USING TTCN-3
Type definitions: boolean, integer, float, bitstring, charstring,
octectstring, hexstring, record, set, enumeration, union
Programming constructs:
message: send/receive
procedure: call/getcall, reply/getreply, raise/catch
if-then-else, loops: for, while, do-while, functions, alternatives
component/port/timer control
Predefined functions:
type conversion, lengthof (string), sizeof (records), …
Overview: e.g. TTCN-3 Quick Reference Card
Eclipse IoT-Testware
THE IOT-TESTWARE PROJECT
20
• Supplement to running and active Eclipse projects
Paho, OM2M, Titan…
• New project at Eclipse Foundation:
https://projects.eclipse.org/projects/technology.iottestware
TTCN-3 test suites for CoAP, MQTT, OPC-UA, LoRa?
• Assured licenses for users
• Current cooperation including
Fraunhofer FOKUS, relayr GmbH, Ericsson, LAAS/CNRS, itemis AG,
Spirent Communications, Easy Global Market, Iskratel/Sintesio, …
THE ECLIPSE PROJECT
21
Take available software and tools …
… and adding public testuites as a result of insights from IoT testing:
IOT-TESTWARE
…
https://projects.eclipse.org/projects/technology.iottestware
22© Fraunhofer FOKUS
IOT QUALITY
1. Conformance
2. Robustness /
Security
3. Performance
Black-Box perspective
23
SAMPLE TESTSUITE STRUCTURE: MQTT
Client
(TS)
Broker
(IUT)
MQTT
Client
(TS)Broker
(IUT)Client
(TS)
cfg_01
cfg_02
Broker
(TS)
Client
(IUT)
cfg_03
Broker as SUT
All mandatory message data fields
Regular and illegal data
(Fixed/variable header, payload)
Protocol features
Connect/disconnect (session)
Subscribe/unsubscribe
Immediate publish
Last will and Testament (LWT)
Heartbeats keepAlive values
Topic
Error handling
Client as SUT
…
24
TEST DEVELOPMENT SAMPLE: MQTT
TESTZIEL-KATALOG
Test configurations
Test Suite Structure
Test purpose (catalogue)
Test implementation (TTCN-3)
25
MQTT BROKER EVALUATION (CONFORMANCE, APRIL 2018)
Broker PASS FAIL INCONCLUSIVE
# % # % # %
HiveMQ ? 39 86,67% 4 8,89% 2 4,44%
Mosquitto 1.4.15 38 84,44% 5 11,11% 2 4,44%
VerneMQ 1.3.1 37 82,22% 5 11,11% 3 6,67%
EMQ 2.2 35 77,78% 8 17,78% 2 4,44%
lannister ? 31 68,89% 12 26,67% 2 4,44%
ActiveMQ 5.15.3 29 64,44% 14 31,11% 2 4,44%
aedes v0.33.0 26 57,78% 17 37,78% 2 4,44%
RSMB ? 26 57,78% 17 37,78% 2 4,44%
RabbitMQ 3.7.4 19 42,22% 26 57,78% 0 0,00%
Mosca 2.8.1 19 42,22% 24 53,33% 2 4,44%
HBMQTT 0.9 17 37,78% 28 62,22% 0 0,00%
Moquette 0.10 16 35,56% 29 64,44% 0 0,00%
26
• Vulnerability scanner:
in particular for web applications, zero-day/fuzzing, consideration of data
bases, traffic/network analyser, program code scanner
• Penetration tester, e.g. “SQL injection”
• Intrusion detection tools
• Load test/Scalability
• Further utilities: Model-based testing (UML testing profile) and risk modelling
TESTWARE: SECURITY
27
FUZZING APPROACH
CoAP
ATS
CoAP
ETSFuzzed
Data
SUT
ATS: Abstract Test Suite
ETS: Executable TS
28
Results for CoAP:
- Initially, 4421 fuzzed test data for CoAP were generated
- After sending the data to a (local) CoAP server, it crashed after date “1107”
https://www.fokus.fraunhofer.de/de/sqc/security_testing
https://github.com/fraunhoferfokus/Fuzzino/blob/master/doc/Fuzzino_XML_Description.pdf
FUZZINO RESULTS AND RESOURCES
29
PERFORMANCE TESTING: HIGH-LEVEL VIEW
Eclipse Titan
RIoT
SUTLoad
htop
30
PERFORMANCE TESTING: WORKFLOW
(1)
(1) Create/choose State Machine
(2) Configure scenario
(3) Start Performance Test Suite
a. Observe RIoT verdicts
b. Observe IUT process with htop
(CPU / Mem. Usage)
(2)
Titan /
RIoT
SUT
htop
(3)
Load
31
IOT-TESTWARE DASHBOARD
Dashboard…
SUTTestdata
Testcontrol
Standardization & Certification
THE IOT-T PROJECT
33
• New Working Group (TST) will develop
IoT test catalogues and specifications (not covered elsewhere)
• The types of testing include
conformance, interoperability, security and performance testing
• The initial technical focus will be:
− IoT network layer
(communication protocols, node connectivity, edge computing etc.),
− Basic security of IoT devices
ETSI TC MTS
34
MTS TST WORK PROGRAMME
IEC 62443-4-2
CoAP
MQTT
LoRaWAN
Vul. database
https://portal.etsi.org/tb.aspx?tbid=860&SubTB=860
35
BASE SECURITY CERTIFICATION SCOPE
The Three Software Stacks Required for IoT Architectures, Eclipse IoT Working Group, September 2016
IoT-Testlab Scope
(basic security level
certification)
telemetry
commands
telemetry
commands
36
THE FOUNDATION
Generic
security requirements
IEC 62443 as basis
Functional AND process
security requirements
Supports domain-specific profiles
IEC 62443
Industrial
IoT
Rail
Medical
Being established as „horizontal standard“
37
IOT PROFILE
Basic IoT security requirements
IEC 62443 subset to define minimum IoT security
IEC 62443-4-2 (component requirements)
Basic IoT security profile
• Focus on SL1
• Excluded aspects:
• Auditing
• Central Security Management
• Secure boot
• DoS- /malicious code protection
Test development
38
SAMPLE SECURITY TEST PURPOSE USING TDL
Foundational TPs, e.g.:
Authenticator feedback
Password lifetime
Session lock
Account changeability
Using TDL-TO, e.g.:
Domain: entities, events, pics
Event sequence
Time label and time constraint
TRIGGER
Expectation
Putting everything together
SUMMARY AND OUTLOOK
40
IOT-TESTWARE
BIG PICTURE
…
ETS
Reporting
Logging
TP: Test Purpose
TSS: Test Suite Structure
ATS: Abstract Test Suite
ETS: Executable TS
SUT: System Under Test
TSS TP ATS
41
Advanced testing technology:
Open source IoT-Testware (code):
External (open source) SW
Standardized IoT test purposes:
SUMMARY
42
• Adding more protocols to IoT-Testware
AMQP, LWM2M, 6LoWPAN, LPWAN
• Increased security level for certification
• Cooperation/liaisons (in preparation) with
ETSI TC Cyber/SmartM2M, oneM2M, OPC Foundation ...
OUTLOOK
