Upload
andy-zaidman
View
2.377
Download
1
Embed Size (px)
Citation preview
Software Analytics: The Dark Side and the Test Side
SEN Symposium 2016 Andy Zaidman, TU Delft
I like to tell stories…
Andy
Jurgen/Frank: Can you give a keynote on
software testing and software analytics at the
SEN Symposium?Me:
Sure I can, I’m honored by the invitation.
Jurgen/Frank: Can you give us a title and
abstract?Me:
<<PANIC>> what should I talk about? Subject so broad...
ARRGHHH...
1. “Software Engineering”
2. “Software Testing”
3. “Software Analytics”
I started by asking
I turned to again
The Dark Side
@ work (Dec. 2015)
http://www.popularmechanics.com/technology/infrastructure/a18721/washington-state-released-3200-inmates-early/
http://townhall.com/tipsheet/mattvespa/2015/12/22/good-news-computer-glitch-caused-the-irs-to-issue-46-million-worth-of-questionable-refunds-n2096374
Bad
http://deredactie.be/cm/vrtnieuws/cultuur%2Ben%2Bmedia/kunsten/1.2490758
vs.
3 - 0
Bring balance, you must!
Test more, you should!
“it depends”
“50% of your time”
What is the (likely) origin of the 50%?
The Mythical Man-Month?Brooks “observed” that testing and debugging took 50% of the time
How? “Calendar research”
When± 40 years ago…
Lack of critical thinking has allowed wrong
claims to take root in the discipline and become
entrenched.
Chinese whispers / telephone game
Folklore/evidence test approximately 50%
Lack of knowledgeHow much time is actually
spent on testing?
NWO Vidi project
Research Tool Software Analytics“Big Brother” in your IDE measures testing activities
• 416 Softw. Engineers, 68 countries • Java / Eclipse • 13.7 man years of development
observed
Immediate feedback for developers
WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on engineering test code versus production code?
50% 50%test code production code
30% 70%test code production code
WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on engineering test code versus production code?
48% - 52%test code production code
After measuring ≥ 3 months
25% - 75%test code
engineering
Most participants overestimate their test engineering activities
production codeengineering
Other observations
Not all Java projects do unit testing
Likely, an underestimation…
No test engineering
For projects with unit test code: 53% of developer did not execute, read, or modify a single test within five months.
Do sets of unit test executions take…
< 5 seconds ≥ 5 seconds
Swift tests
Execution of set of unit tests takes 0.54 seconds (median).
Ad hoc test selection
Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.
Tests fail
65% of test executions in the IDE fail.
Test Driven Development
TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.
Software Analytics?
What about
Testing Analytics?What about
Software analytics =
bait / intrinsic motivation
From an experienced developer…“Estimated time working on tests: 20%,
actual time: 6%. Cool statistics! ”
Testing analytics helps developers to get a better understanding about their own
development behavior
Hawthorne effect?
Hawthorne effect?“Keeping up appearances” > 3 months?
Takeaways
Triangulate your results mixed methods research
Be critical towards “folklore” Core might be true, but is it (still) valid,
valid in your context, etc.
Don’t talk testing, but act testing
Takeaways
Critique?• Only developer testing ( ≠ Brooks)• Only Java, only Eclipse• Only 416 developers • Only inside the IDE
Extending to IntelliJ and > 1500 developers
Also looking into Continuous Integration
Software engineering
folklore
Perception w.r.t. dev. testing
Measured developer testing
48% - 52%
25% - 75%
Software analytics help engineers to better understand
their own behavior
azaidman
WatchDog Team
References• M. Beller, G. Gousios, A. Panichella, A. Zaidman. When, How and Why
Developers (Do Not) Test in Their IDEs. Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 179-190, 2015.