An introduction and future of Ruby coverage library

RubyKaigi 2017 (19th Sep. )

Yusuke Endoh (@mametter)

Yusuke Endoh (@mametter)

• Ruby committer (2008—)

• Full-time Ruby committer (2017—)

• My goal: make Ruby programs robust

– Test coverage, and type system?

Method.Put the cream cheese into the mixing bowl.Put the sour cream into the mixing bowl.Put the white sugar into the mixing bowl.Put the yogrut into the mixing bowl.Put the unsalted butter into the mixing bowl.Combine the milk.Put the cake flour into the mixing bowl.Combine the corn starch.Put the brown sugar into the mixing bowl.Combine the egg whites.Combine the egg yolks.Put the lemon juice into the mixing bowl.Stir for 7 minutes.Liquify the contents of the mixing bowl.Pour contents of the mixing bowl into the baking dish.Bake the cake mixture.Watch the cake mixture until baked.

Serves 4.

Cheese cake in Chef.

Ingredients.100 g cream cheese97 g sour cream107 g yogrut112 g white sugar11 g brown sugar37 g unsalted butter37 g cake flour3 g corn starch3 ml milk3 egg yolks3 egg whites10 ml lemon juice0 g cake mixture

Cooking time: 80 minutes.

‘d’‘a’

‘k’‘p’

11*3*3=99 ‘c’

37*3=111 ‘o’

¥n

‘o’

Push characters‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’

into the stack

Convert them to a stringand “serve” it.

data

code

Esoteric Recipe

• Polyglot of Chef andreal recipe

– Japanese versionhttps://cookpad.com/recipe/4649810

– English versionhttps://cookpad.com/us/recipes/3335222

My main contributions for Ruby

• Implementation of some features:keyword arguments, deadlock detection, etc.

• Release management for Ruby 1.9.2 and 2.0.0

• Optcarrot: A NES emulator for Ruby3x3 benchmark

• Enhancement ofthe test suite of Ruby

• coverage.so: the core libraryfor coverage measurement

’06B ’07A ’07B ’08A60

70

80

90

100

covera

ge (

%)

70%

85%

line coverage

Today’s theme

• An introduction of test coverage

• An improvement plan of Ruby’s coverage measurement feature towards 2.5

Survey [1/3]

• Q. Do you use Ruby/RoR in production?

– Raise your hand, please!

Survey [2/3]

• Q. In those, do you test your code?

Survey [3/3]

• Q. In those, do you use “coverage”?

– Do you check the result of SimpleCov?

Agenda

• What is coverage

• How to understand and use coverage

• The current status of Ruby coverage feature

• The future plan of Ruby coverage feature

• Conclusion

Agenda

☞ What is coverage

• How to understand and use coverage

• The current status of Ruby coverage feature

• The future plan of Ruby coverage feature

• Conclusion

What is coverage?

• A measure of “goodness” of a test suite– Also called “test coverage” or “code coverage”

• Allows you:– Find untested code– Decide whether your test suite is good enough

or not yet• (This is arguable, but I think we can use it as an

advice)

• Types of coverage– Function coverage, line coverage, branch

coverage, …

Function coverage

• How many functions are executed by the tests# codedef foo; …; end # ✓def bar; …; end # ✗def baz; …; end # ✓

# testfoobaz

2/3(67%)

• Advantage• Easy to understand

• Easy to visualize

• Disadvantage• Too weak as a measure

Line coverage

• How many lines are executed

# codedef foo(x) # ✓

if x == 0 # ✓p :foo # ✗

elsep :bar # ✓

endend

# testfoo(1)

3/4(75%)

Non-significant line isignored

• Advantage• Easy to understand• Easy to visualize

• Disadvantage• Still weak as a measure

• foo() if x == 0

Branch coverage

• How many branches are taken true and false

# codedef foo(x)

p :foo if x == 0 # ✓p :bar if x < 2 # ✗

end

# testfoo(0)foo(1)

1/2(50%)

• Advantage• Relatively exhaustive

• Disadvantage• Difficult to visualize

true-case and false-case areboth executed

Coverage types

Coverage type Easy tounderstand/

visualize

Exhaustive

Functioncoverage

○ ✕

Linecoverage

○ △

Branchcoverage

△ ○

Currently, Ruby supports only line coverage

Other types of coverage

• Condition coverage– How many conditions

(not branches) are takenboth true and false

• Path coverage– How many paths are executed

– Combinatorial explosion

• Other advanced ones– Data-flow coverage

– MC/DC coverage

if a && b

branch

conditioncondition

Trivia

• “C0/C1/C2 coverages” have difference meanings to different people

– C0 coverage = line coverage

– C1 coverage = branch coverage or path coverage?

– C2 coverage = condition coverage or path coverage?

Coverage and Ruby

• In Ruby, Coverage is crucial!– A test is the only way to ensure quality– Coverage is important to measure test goodness

• Considering it, coverage is not used so much…– Coverage is not well-known?– It is not well-known how to use coverage?– Ruby’s coverage measurement feature is not

enough?

• I’d like to improve the situation with this talk

Agenda

• What is coverage

☞ How to understand and use coverage

• The current status of Ruby coverage feature

• The future plan of Ruby coverage feature

• Conclusion

What is a good test suite?

• Covers the code

– Coverage measures this

• Also covers the design of program

– Coverage does not measure this directly

How to understand coverage

• Coverage is just a measure

• Coverage is not a goal

If coverage XX% is required as a goal…

• Developers will1. Pick untested code that looks easiest to

cover

2. Write a trivial test just for the code

3. Iterate this procedure until XX% is achieved

• It will result in trivial, not-so-good test suite– It may be exhaustive for the code itself, but

– It won't be exhaustive for the design

A good way to improve coverage

• Developers should1. Look at untested code

2. Consider what “test design” is insufficient

3. Write them

– In consequence of them, the untested code is executed

• It will result in good test suite– It will be exhaustive not only for the code

but also for the design

How many % is needed/enough?

• It depends upon the module being tested– Aim 100% for a significant module (e.g., injure

someone)

– Don't worry too much for a non-significant module

• It also depends upon cost performance– For non-significant module, write a test only if it

is not so hard

• Again: Coverage is not a goal

Agenda

• What is coverage

• How to understand and use coverage

☞ The current status of Ruby coverage feature

• The future plan of Ruby coverage feature

• Conclusion

Ruby's coverage ecosystem

• SimpleCov

• coverage.so

• Concov

SimpleCov

• A wrapper library for coverage.so• Visualization with HTML• Useful features: merging, filtering, for Rails app• Author: Christoph Olszowka (@colszowka)

Usage of SimpleCov

• Write this at the top of test/test_helper.rb

• Run the test suite

• coverage/index.html will be produced

– Note: SimpleCov cannot measure already-loaded files before SimpleCov.start

require "simplecov"SimpleCov.start

coverage.so

• The only implementation of coverage measurement for Ruby 1.9+

• SimpleCov is a wrapper for coverage.so

• Author: me

Basic usage

# test.rbrequire "coverage"Coverage.start

load "target.rb"

p Coverage.result#=> {"target.rb"=># [nil,nil,1,1,1,nil,# 0,nil,nil,nil,1]}

Start measuring coverage

Load the target file

Stop measuringand get the result

Coverage data

Coverage data

# target.rb

def foo(x)if x == 0

p 0else

p 1end

end

foo(1)

[nil,nil110

nil1

nilnilnil1]

nil:Non-significant line

Number:Count executed

Untested line!

Method definition is code in Ruby

# target.rb

def foo(x)if x == 0

p 0else

p 1end

end

[nil,nil100

nil0

nilnil]

Method definition iscounted as an

execution

(It is not a count ofmethod invocation!)

I regret the design of coverage.so

• Support only line coverage• Excuse: I introduced it just for test

enhancement of Ruby itself– I didn't deliberate the API for external

project…

• I have wanted to make the next version better

ext/coverage/coverage.c:

69 /* Coverage provides coverage measurement feature for Ruby.70 * This feature is experimental, so these APIs may be changed in future.71 *

Concov

• CONtinuous COVerage– Detects temporal change (degradation) of

coverage

• Developed for monitoring Ruby's coverage

• Author: me

• Presented at RubyKaigi 2009, and then…– It has not been used by everyone (including me)

– It was based on Ramaze (old web framework)!

Concov reveals reality of Ruby dev.（Enumerable#join, 2009/07/07, M***）

New feature

introduced

with no tests!

Concov reveals reality of Ruby dev.（File#size, 2009/02/25, M***）

Concov reveals reality of Ruby dev.（Etc::Passwd.each, 2009/02/19, N*****）

Concov reveals reality of Ruby dev.（Dir.home, 2009/02/03, N*****）

Concov reveals reality of Ruby dev.（Array#sort_by!, 2009/02/03, M***）

Concov reveals reality of Ruby dev.（rb_to_float, 2008/12/30, M***）

Coverage ecosystem for other languages

• C/C++: GCOV/LCOV– Integrated with gcc:gcc -coverage target.c

• Java: A lot of tools– Cobertura, Emma,

Clover, JaCoCo– Integrated with CI tools

and/or IDEs

• JavaScript: IstanbulJenkins Cobertura plugin

LCOV result

Agenda

• What is coverage

• How to understand and use coverage

• The current status of Ruby coverage feature

☞ The future plan of Ruby coverage feature

• Conclusion

A plan towards Ruby 2.5

• Support function and branch coverage

– There have been multiple requests and some PoC patches…

• To make the API better, any comments are welcome

– https://bugs.ruby-lang.org/issues/13901

API: to start measuring

# compatible layerCoverage.startCoverage.result#=> {"file.rb"=>[nil, 1, 0, …], … }

# new APICoverage.start(lines: true)Coverage.result#=> {"file.rb" => { :lines => [nil, 1, 0, …] } }

API: to start other types of coverage

# enable branch and function coverageCoverage.start(lines:true,

branches:true,methods:true)

Coverage.result#=> {"file.rb" => { :lines => [nil, 1, 0, …],# :branches => {…},# :methods => {…} } }

# shorthandCoverage.start(:all)

Coverage.result for branch coverage

{"target1.rb"=>{:lines=>[…],

:branches=>{[:if, 0, 2]=>{

[:then, 1, 3]=>2,[:else, 2, 5]=>1

}},:methods=>{

[:test_if, 1]=>3}}}

# target1.rb1: def test_if(x)2: if x == 03: p "x == 0"4: else5: p "x != 0"6: end7: end8: 9: test_if(0)10: test_if(0)11: test_if(1)

From if at Line 2

Jumped tothen clauseat Line 3

twice

Jumped toelse clauseat Line 5

once

Coverage.result for branch coverage

{"target2.rb"=>{:lines=>[1, 1, 1, nil, nil, 1],

:branches=>{[:if, 0, 2]=>{

[:then, 1, 2]=>1,[:else, 2, 2]=>0},

[:if, 3, 3]=>{[:then, 4, 3]=>0,[:else, 5, 3]=>1}},

:methods=>{[:test_if, 1]=>3

}}}

# target2.rb1: def test_if_oneline(x)2: p "x == 0" if x == 03: p "x != 0" if x != 04: end5: 6: test_if_oneline(0)

Line coverage 100%

Branch coverage tells youthere are untested cases

Discussion of API design

• 100% compatible

• [<label>, <numbering>, <line-no>]– e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1]– <numbering> is a unique ID to avoid conflicts for the

case where there are multiple branches in one line• LCOV-style

– Other candidates:• [<label>, <line-no>, <column-no>]

– How to handle TAB character

• [<label>, <offset from file head>]– Looks good, but hard to implement (I'll try later)

Overhead of coverage measurement

(just preliminary experiment)

# Example 21: foo()2: foo()…

99: foo()100: foo()

Benchmark w/o cov. w/ cov. Overhead

Example 1 0.322 μs 6.21 μs x19.3

Example 2 1.55 μs 7.16 μs x4.61

make test-all 485 s 550 s x1.13

# Example 11: x = 12: x = 1…99: x = 1100: x = 1

Demo

• Applied the new coverage.so to Ruby

• Integrated with C code coverage by GCOV and Visualized by LCOV

Ruby codein stdlib

make exam withgcc -coverage

make examCOVERAGE=1

test-coverage

.dat

*.gcda gcov

my script

run test

C codeof MRI

cov. datasource aggregate

lcov HTML

Jenkins Cobertura Plugin

Agenda

• What is coverage

• How to understand and use coverage

• The current status of Ruby coverage feature

• The future plan of Ruby coverage feature

☞ Conclusion

Acknowledgement

• @_ko1• @t_wada• @kazu_cocoa• @moro• @makimoto• @dev_shia• @tanaka_akr• @nalsh• @spikeolaf• @k_tsj

Conclusion

• What is coverage, how important in Ruby, and how to understand coverage

• The current status of Ruby's coverage measurement and ecosystem

• A plan towards Ruby 2.5 and preliminary demo– Any comments are welcome!

– https://bugs.ruby-lang.org/issues/13901

Future work

• Determine the API

• define_method as method coverage

• &. as branch coverage

• Callsite coverage

• Block coverage

obj.foo.bar

ary.map { …… }