2. Optimizing Java Device Testing with Automatic Feature
Discover y Fyodor Romanov, Software Engineer 3. Agenda
- Java device testing challenges.
4. Testing framework structure. 5. Approach how to optimize and
shorten device test-cycle using Diagnostic tests to explore device
capabilities. 6. Demo showing Diagnostic tests usage and benefits
7. Problem description 8. Problem description
- Nearly 3 Billion devices run Java ME in the world today.
9. Before a manufacturer can make a device commercially
available, they must perform a significant amount of testing to
ensure a high-quality, reliable product that delivers customer
satisfaction. 10. Java Device quality testing helps Java technology
wireless device manufacturers and network operators maximize
product quality and minimize time to market. 11. Problem
description There are a lot of (more then 30) Java Platform, Micro
Edition (Java ME platform) technologies (and combinations): 12.
Problem description
- A lot of technologies, a lot of tests. More then 11,000 tests
are developed currently in JDTS and number is growing.
13. Up to 3 weeks for guru to run all test on one device. 14.
Challenge of managing and, if possible, lowering quality testing
costs. 15. The most challenging task for test execution is to
configure tests for particular device. 16. Express testing:
evaluate device quality in 4 hours. 17. Problem description
- Not to test technologies which device does not support.
- Negative false cases complicate failure analysis
18. Problem statement Device manufacture knows, but testing
house doesn't. 19. Before Automatic feature discovery. 20. Solution
in nutshell 21. Solution implementation 22. Solution implementation
Implementation example will be based onJDTF test frameworkhowever
it should be applicable to any Java device testing framework. The
Java Device Test Framework (JDTF) is a test framework based
onOracle s commercialJava Device Test Suite (JDTS)product. JDTF is
a general purpose, fully-featured, flexible, and configurable test
framework suited to assess various aspects of Java Platform, Micro
Edition (Java ME) device implementation quality . 23. JDTF overall
tests structure JDTF organizes tests intohierarchical structure s
calledtest packs ,a collection of tests that are functionally
related and have common setup requirements : 24. JDTF testing
structure Harness is using tests and different configuration files
to execute test on device. Test execution on device is controlled
by MicroAgent which starts/stops test and communicates with Harness
(sends results, logs, etc.) 25. Tests execution workflow 26. How to
distinguish diagnostic tests. JDTF provides keywordsto
filter(subset for execution) tests intest packs. So two options
available to distinguish diagnostic tests:
- Mark them with special keyword, for example diagnostic
27. Move them into special dedicated test pack. 28. Writing
diagnostic test. Changes in MicroAgent. Diagnostictests find out
whether the device supports an optional feature and export a
property by usingLogger classas follows. public class Logger
{...public void exportProperty(String name, String value) {
LogRecord rec = new LogRecord (...);
rec.setProperty(EXPORTED_PROPERTY_PREFIX + name, value); ...} } 29.
Writing diagnostic test
- Test name should clear indicate what optional feature is tested
and placed in appropriate package, for
example:ConnectionTest.testHTTP
30. Test should be marked with diagnostic keyword 31. Test
should try to use optional feature and
callLogger.exportProperty(String name, String value) 32. If feature
is supported, return PASSED and if feature is not supported return
FAILED. This helps to visualize test result. 33. Do nothing else
34. Writing diagnostic test. Code Sample. public class
ConnectionTest implements AutomatedTest {...public boolean
testHTTP() { Connection c = null; try {... c = Connector.open(url);
logger.exportProperty("HTTPSupport", "true"); return true; } catch
(IllegalArgumentException iae) {return false; } ...} 35. Developing
Template Generator 36. Adding Template Builders to TestPacks 37.
Changes in FW 38. Final touch. Adjusting Relevance Filters
Relevance filter should filter out not relevant test providing
right reason. Code sample: public String
LapiDependencyProvider.isRelevant(final TestId id) throws
TestDependencyProviderException { final boolean
createDeleteLmsSupported
=getBoolean(TestPackProperty.CreatingAndDeletingLandma
rkStoresSupported, id); if (!createDeleteLmsSupported) { if
(fqn.endsWith("#addLandmarkScenario2")
||fqn.endsWith("#deleteLandmarkFromAnotherLMS") { return "Test not
relevant because Landmark Store create/delete operations configured
to be not supported."; } } 39. Automatic Feature Discovery
Efficiency.
- Proved to be very valuable.
40. Reduces 1 - 2 days work for one person (depending of
person's experience) to 30 minutes. 41. BeforeAutomatic Feature
Discovery feature customerswere taking a complete unknown device
and painstakingly reading logs to find out what itsupports . 42.
DEMO 43. Useful links
- JDTF Project home pagehttps://jdtf.dev.java.net with source
code
44. JDTS home page with API detailed description 45.
http://www.oracle.com/technetwork/java/javadevice-140362.html 46.
Session k ey points
- HowDiagnostictests can help in Java
device quality testing
- Tips how to develop such tests
47. How to processDiagnostictests output in testing
frameworkJDTF test framework wasused for demonstration 48. Q&A
Q&A 49.
