Sikuli-SlidesKhalid AlharbiSikuli Lab

Outline

• Introduction.• What’s Sikuli.• What’s Sikuli-Slides.• Sikuli-Slides for automated GUI testing.• Sikuli-Slides for interactive tutorials.• Conclusions.

GUI Automation

GUI Testing

Click 400, 300Type “Alan”

Click ‘first_name’Type “Alan”

What’s Sikuli?

How Sikuli works?

How Sikuli works?

Example 1GUI Automation

Example 2 GUI Testing

Why Sikuli?

• Uses Computer Vision.• No scripting API support or source code

access.• Interacts with anything you see on the

screen.• OCR support.• Works on Web-based UIs.• Works on virtual machines and remote

desktops

Sikuli Project

• Sikuli Script• Sikuli Java API• Sikuli Slides

Sikuli script

Sikuli Java API

Sikuli-Slides

Sikuli-Slides

• Automate and test GUIs by using screenshots and annotating them.

• Make visual automation accessible to everyone.

• Use a tool that most users already know how to use.

• Reinventing Computer-based tutoring.

Motivation

• PowerPoint is already a popular tool for creating test cases.

• Online tutorials that include annotated screenshots.

How to tell computers how to interact with applications?

How to tell users how to use applications?

Why PowerPoint

• Most users already know how to use PowerPoint.

• Office Open XML file format a.k.a OOXML.• DrawingML• Shapes, pictures, etc.

• Data Interoperability.

System Architecture

Document Parser Visual

Automation Processor

Java API

C++ Engine

OpenCVjava.awt.Robot

PowerPoint

Document (.pptx file)

How to represent user input actions?

Action Shape

Left click Rectangle

Right click Oval

Double click Frame

Keyboard typing Text Box

Open default browser Cloud

Drag and drop Rounded Rectangle connected by an arrow pointing to the drag and drop direction.

Text ……..www.sikuli.or

g

Demo

But using special shapes is bad!

Click

Click

Demo

How to identify identical targets?

Click

Click

Click

Click

Click

Click

Right

Top

Bottom

Left

Demo

Sikuli Slides Applications

• GUI Testing• Tutorials

How to test a GUI in sikuli-slides?

Sikuli-Slides makes GUI testing accessible to all QA engineers.

GUI Testing

Demo

How to learn about a software?

Tutorials

Sikuli Slides can run live interactive tutorials

But where’s the audio?

Adding audio or narration

• How to add audio narrations to the slides and sync the audio with the GUI input action to add more interactive experience when running the slides?

• How to annotate the screen with text so users can layer the screen with informative text that explains what’s running on the screen?

Demo

Where are we now?

• Support three modes in which you can run presentation slides in:• Action mode. • Tutorial mode.• Development mode.

Conclusions

• Sikuli-Slides:• Uses Computer Vision.• Can run presentation slides on

Windows/Mac/Linux.• Makes visual automation accessible to

everyone.• Features a new way to create computer based

tutorials.

Thanks!

• http://code.google.com/p/sikuli-api/wiki/SikuliSlides

• Or just google “sikuli slides”

• slides.sikuli.org