Minitest 6: Test Feistier! Ryan Davis, Seattle.rb Minitest 6 · Presentation Sammich 7. Minitest 6: Test Feistier! RailsConf 2018, Pittsburgh, PA Ryan Davis, Seattle.rb

Minitest 6: Test Feistier!

RailsConf 2018, Pittsburgh, PA

Ryan Davis, Seattle.rb

@the_zenspider





@the_zenspider

Setting ExpectationsOverview / Idea TalkLittle CodeTesters at All Levels

265 slides / 40 min = ~7 spm�2




@the_zenspider

🎺🎷

🎊🎉�3




@the_zenspider

1000+ Gems!

�4




@the_zenspider

Narrative?

�5




@the_zenspider�6




@the_zenspider

%?

�6




@the_zenspider

%?🥙?

�6




@the_zenspider

%?🥙?🍔?

�6




@the_zenspider

what you’ll tell themActual Content

what you told them

Presentation Sammich

�7




@the_zenspider

😞

�8




@the_zenspider

➡🍔💬 = 👎

�9




@the_zenspider�10




@the_zenspider

🗣🍔💬 = 🤢

�11




@the_zenspider�12

https://www.flickr.com/photos/hsincheng_lin/12180213616




@the_zenspider

🥅

�13




@the_zenspider

Sorry! 😨

�14




@the_zenspider

💍

�15




@the_zenspider

🤷2

�16




@the_zenspider

🖐Audience🖐

�17




@the_zenspider

Beginner?

�18




@the_zenspider

Minitest User?

�19




@the_zenspider

RSpec User?

�20




@the_zenspider

RSpec Fanatic?

�21




@the_zenspider

All are Welcome!

�22




@the_zenspider

(I Heckle Back)

�23




@the_zenspider

My Baby: Minitest

�24




@the_zenspider

❤

�25




@the_zenspider

Testing=Passion

�26




@the_zenspider

1995!

�27




@the_zenspider

Pre-XP

�28




@the_zenspider

TDDesign

�29




@the_zenspider

TDDevelopment

�30




@the_zenspider

TDD+YAGNI

�31




@the_zenspider

Better Software

�32




@the_zenspider

Every Time�33




@the_zenspider

Every Time�33

(Almost)




@the_zenspider

Minitest

�34




@the_zenspider

What I am talking about

�35




@the_zenspider

Introduction

�36




@the_zenspider

Quick History

�37




@the_zenspider

Philosophy &

Purpose�38




@the_zenspider

Major Changes

�39




@the_zenspider

Architectural

�40




@the_zenspider

& API Changes

�41




@the_zenspider

What I am not talking about

�42




@the_zenspider

Testing=Good

�43




@the_zenspider�44




@the_zenspider�44




@the_zenspider

Use Minitest?

�45




@the_zenspider

✅Fantastic!

�46




@the_zenspider

❌That’s OK!

�47




@the_zenspider

Informative

�48




@the_zenspider

🤜rspec🤛

�49




@the_zenspider�50




@the_zenspider�51

�i

∞

meh




@the_zenspider

∀x: minitest(x) < rspec(x)

�52




@the_zenspider

∀x: minitest(time) < rspec(time)

�53




@the_zenspider

∀x: minitest(mem) < rspec(mem)

�54




@the_zenspider

∀x: minitest(users) < rspec(users)

�55




@the_zenspider

∀x: minitest(dl) < rspec(dl)

�56




@the_zenspider

At the end of the day

�57




@the_zenspider

I do NOT care, as long as you test.

�58




@the_zenspider

That Said...

�59




@the_zenspider

Minitest is faster and more direct

�60




@the_zenspider

⚙

�61




@the_zenspider

What & Why

�62




@the_zenspider

∂�63




@the_zenspider

Guided Read of Minitest

�64

http://confreaks.tv/videos/rubyconf2015-a-guided-read-of-minitest




@the_zenspider

Writing a Test Framework

from Scratch�65

http://www.zenspider.com/presentations/2016-roa.html




@the_zenspider

what you’ll tell them

Presentation Sammich

�66




@the_zenspider

Quick Introduction

�67




@the_zenspider

Test Framework

�68




@the_zenspider

Unit & Functional

�69




@the_zenspider

XUnit-StyleClassesMethods

Assertions�70




@the_zenspider

Spec-Styledescribe blocks

it blocksexpectations

�71




@the_zenspider

HardwareAgnostic

Performance Testing

�72




@the_zenspider

Stubs & Mocks

�73




@the_zenspider

TDD & BDD

�74




@the_zenspider

Clean Architecture

�75




@the_zenspider

Very Very Fast

�76




@the_zenspider

Flexible Reporting

�77




@the_zenspider

Extensible: 200+ Gems

�78




@the_zenspider

The Briefest History of Minitest

�79




@the_zenspider

😴

�80




@the_zenspider

2004 test/unit

�81




@the_zenspider

🤷

�82




@the_zenspider

22 files, ~2400 LOC, ~2500 flog

�83




@the_zenspider

5 Runners

�84




@the_zenspider

2 “Collectors”

�85




@the_zenspider

and ALL the pattern buzzwords

�86




@the_zenspider

template engine

�87




@the_zenspider

assert_block

�88




@the_zenspider

Why?

�89




@the_zenspider

Failure Driven

Development�90




@the_zenspider

~25 Projects

�91




@the_zenspider

Best of All

�92




@the_zenspider

Only 75 lines!

�93

at_exit { Test::Unit.autotest }

module Test class Assertion < Exception; end class Unit VERSION = "1.0.0" def self.autotest ObjectSpace.each_object(Class) do |klass| next unless klass < Test::Unit::TestCase inst = klass.new klass.public_instance_methods(true).each do |meth| next unless meth.index("test") == 0 begin inst.setup inst.send meth.intern inst.teardown rescue Exception => e print "\n", (Test::Assertion === e ? "Failure: " : "Error: ") puts "#{klass}.#{meth}: #{e}" puts e.backtrace end end end end end end

module Test class Unit class TestCase def setup; end def teardown; end def assert(test, msg="failed assertion (no message given)") raise Test::Assertion, msg unless test end def assert_equal(exp, act, msg=nil) assert exp == act, msg || "Expected #{act.inspect} to be equal to #{exp.inspect}" end def assert_in_delta(exp, act, delta, msg=nil) assert((exp.to_f - act.to_f).abs <= delta.to_f, msg || "Expected #{exp} to be within #{delta} of #{act}") end def assert_instance_of(cls, obj, msg=nil) assert cls === obj, msg || "Expected #{obj} to be a #{cls}" end def assert_kind_of(cls, obj, msg=nil) assert obj.kind_of?(cls), msg || "Expected #{obj.inspect} to be a kind of #{cls}" end def assert_match(exp, act, msg=nil) assert act =~ exp, msg || "Expected #{act.inspect} to match #{exp.inspect}" end def assert_nil(obj, msg=nil) assert obj.nil?, msg || "Expected #{obj.inspect} to be nil" end def assert_not_equal(exp, act, msg=nil) assert exp != act, msg || "Expected #{act.inspect} to not be equal to #{exp.inspect}" end def assert_not_nil(obj, msg=nil) assert ! obj.nil?, msg || "Expected #{obj.inspect} to not be nil" end def assert_not_same(exp, act, msg=nil) assert ! exp.equal?(act), msg || "Expected #{act.inspect} to not be the same as #{exp.inspect}" end def assert_raises(exp, msg=nil) yield assert false, "Expected #{exp} to be raised" rescue Exception => e assert exp === e, msg || "Expected #{exp} to be raised, but got #{e.class}" return e end alias :assert_raise :assert_raises def assert_same(exp, act, msg=nil) assert exp.equal?(act), msg || "Expected #{act.inspect} to be the same as #{exp.inspect}" end def assert_operator(o1, op, o2, msg="") assert o1.__send__(op, o2), msg || "Expected #{o1}.#{op}(#{o2}) to be true" end def assert_nothing_raised; yield; end end end end




@the_zenspider

History in a 🥜

�95

2006-10 1.0.0 Initial Release.2008-06 1.2.0 Full Assertions, Specs, and Mocks.2010-11 2.0.0 Benchmarks.2012-05 3.0.0 Stubs & other enhancements.2012-09 4.0.0 Nothing external. Internal name changes.2013-05 5.0.0 Huge overhaul. What I really wanted.2018-01 5.11.3 Current functionality. Expanded Reporters.




@the_zenspider

History in a 🥜

�95





@the_zenspider

History in a 🥜

�96





@the_zenspider�97




@the_zenspider�97




@the_zenspider

Complexity Over Time

�98

0

750

1500

2250

300020

06

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

1 1.2 2 34 5 5.116

VersionLOCFlog




@the_zenspider

0

4500

9000

13500

18000

2.0 3.0 3.5

FlogLOC rspec 2-3.5

�99




@the_zenspider

0

4500

9000

13500

18000

2.0 3.0 3.5

FlogLOC

minitest

rspec 2-3.5

�99




@the_zenspider

Minitest Firsts

�100




@the_zenspider

Shuffle tests by default

�101




@the_zenspider

Test & Spec, TDD & BDD

�102




@the_zenspider

Benchmark Tests

�103




@the_zenspider

Purpose & Philosophy of

Minitest�104




@the_zenspider

I ❤ XP

�105

http://www.projectcartoon.com




@the_zenspider

Test First

�106

http://wiki.c2.com/?CodeUnitTestFirst




@the_zenspider

DTSTTCPW “dut-st-t-cupwuh”???

�107

http://wiki.c2.com/?DoTheSimplestThingThatCouldPossiblyWork




@the_zenspider

YAGNI

�108

http://wiki.c2.com/?YouArentGonnaNeedIt




@the_zenspider

Incredibly Simple

�109




@the_zenspider

Learning Ruby is hard enough!

�110




@the_zenspider

🏃Takes a Lot=

�111




@the_zenspider

minitest? rspec? Beginner?

Magic!�112




@the_zenspider

Ruby is Magic

�113




@the_zenspider

def is my stub frameworkthingy = Something.new

def thingy.expensive_value 42 end

# …write fast tests…

�114




@the_zenspider

Less Magic

�115




@the_zenspider

Framework < Learning

�116




@the_zenspider

Expand Toolbox

�117




@the_zenspider

Lower the Bar

�118




@the_zenspider

Lower the BarClasses & Objects

�118




@the_zenspider

Lower the BarClasses & ObjectsMethods & Method calls

�118




@the_zenspider

Lower the BarClasses & ObjectsMethods & Method callsReading API Doco

�118




@the_zenspider

Nothing Extra

�119




@the_zenspider

Usable Tools

�120




@the_zenspider

Minimal Testrequire "minitest/autorun" class TestCalculator < Minitest::Test def test_basics assert_equal 4, calculate("2+2") end end

�121




@the_zenspider

Minimal Specrequire "minitest/autorun" describe Calculator do it "should handle the basics" do _(calculate "2+2").must_equal 4 end end

�122




@the_zenspider

1:1

�123




@the_zenspider

specs are tests

�124




@the_zenspider

Just Ruby™

�125




@the_zenspider

Just Use Ruby™

�126




@the_zenspider

Incredibly Fast

�127




@the_zenspider

Fastest Test

Framework�128




@the_zenspider

It achieves that by doing nothing extraMinitest.autorun Minitest.run(args) Minitest.__run(reporter, options) Runnable.runnables.each runnable.run(reporter, options) self.runnable_methods.each do |method| self.run_one_method(method, reporter) Minitest.run_one_method(klass, method) klass.new(runnable_method).run

�129




@the_zenspider�130




@the_zenspider

(Almost) Nothing Extra

�131




@the_zenspider

Store Nothing Extra

�132




@the_zenspider

GC

�133




@the_zenspider

Don’t Create Extra Objects

�134




@the_zenspider

Release ASAP

�135




@the_zenspider

Memory = #tests

�136




@the_zenspider

Memory = #failures

�137




@the_zenspider

rspec is nx

�138




@the_zenspider

Pain = Slow

�139




@the_zenspider

RSpec ~30% slower on a

fast suite�140




@the_zenspider

Improve Testing

�141





Randomize Test Order

Require Assertion Call?

Incre

dibly

Fast

Tests

Parallelization by default

01

2

3

45

6 78

9

1011

Only meaningful assertions

Choice: tests or specs

Benchmark Tests

Test Driven Design Refac

tor A

sser

tions

Refactor Test Classes

???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


???




@the_zenspider

Reproducibility

�154




@the_zenspider

Reproducibility is

Paramount�155




@the_zenspider

--seed ###�156




@the_zenspider

--name /pattern/

�157




@the_zenspider

i_suck_and_my_tests_are_order_dependent!

�158




@the_zenspider

No (less?) Dogma

�159




@the_zenspider

I ❤ That!

�160




@the_zenspider

Not for Everyone

�161




@the_zenspider

Should It?

�162




@the_zenspider

For me:

�163




@the_zenspider

Test First (red, green, refactor)

�164




@the_zenspider

DTSTTCPW

�165

(do the simplest thing that could possibly work)




@the_zenspider

Mock Last

�166








@the_zenspider

Simple.

�167




@the_zenspider

Simple.Statist.

�167




@the_zenspider

Simple.Statist.Tests.

�167




@the_zenspider

“You should write a test for anything that could break”

—Kent Beck

�168




@the_zenspider

Major MT5 Changes

�169




@the_zenspider

Major namespace rename MiniTest::Unit::TestCase

-> Minitest::Test.

�170




@the_zenspider

Added Minitest::Benchmark

�171




@the_zenspider

No Managers No Runners

�172





MiniTest

MiniTest::Assertion

MiniTest::Unit::Guard

MiniTest::Assertions

MiniTest::Expectations

MiniTest::Unit::TestCase

MiniTest::Spec::DSL

MiniTest::Unit::LifecycleHooks

MiniTest::PrideIO

MiniTest::Unit

MiniTest::PrideLOL

MiniTest::Spec

MiniTest::Skip

Minitest::BacktraceFilter Minitest::ParallelEach Minitest::Mock

Minitest 4





Minitest

Minitest::Runnable

Minitest::Reporter Minitest::Assertion

Minitest::Guard

Minitest::BacktraceFilter

Minitest::Assertions

Minitest::Expectations

Minitest::Test

Minitest::Benchmark Minitest::Spec::DSL

Minitest::Test::LifecycleHooks

Minitest::ParallelEach

Minitest::PrideIO

Minitest::PrideLOL

Minitest::BenchSpec

Minitest::Spec

Minitest::Skip Minitest::UnexpectedErrorMinitest::CompositeReporter

Minitest::Mock

Minitest 5





MiniTest

Minitest::Runnable

Minitest::Reporter Minitest::Assertion

Minitest::Guard

Minitest::BacktraceFilter

Minitest::Assertions

Minitest::Expectations

Minitest::Test

Minitest::Benchmark Minitest::Spec::DSL

Minitest::Test::LifecycleHooks

Minitest::ParallelEach

Minitest::PrideIO

Minitest::PrideLOL

Minitest::BenchSpec

Minitest::Spec

Minitest::Skip Minitest::UnexpectedErrorMinitest::CompositeReporter

Minitest::Mock

MiniTest::Unit

MT4 -> MT5Minitest




@the_zenspider

Major MT6 Changes

�176




@the_zenspider

Architectural Changes in Minitest 6

�177




@the_zenspider

Same Namespace!

�178




@the_zenspider

Same Classes!

�179




@the_zenspider

No Cruft!

�180




@the_zenspider

5.12.0 MT5 w/ deprecations on. 5.11.x MT5 w/o deprecations.5.3.3 Un-shuffled test suites.

Compatibility Chart

�181




@the_zenspider

Yeah Yeah

�182




@the_zenspider

Parallelism

�183




@the_zenspider

parallelize_me!

�184




@the_zenspider

Make parallel the default?

�185




@the_zenspider

serialize_me!

�186




@the_zenspider

Testing @ Scale Implementations

That Scale�187




@the_zenspider

All Tests, Completely

Shuffled�188




@the_zenspider

Distributed Testing

�189




@the_zenspider

Distribute Locally

�190




@the_zenspider

Use all your CPUs!

�191




@the_zenspider

jeremyevans/minitest-parallel_fork 2 forks 4 forks spec_core: 1.25x - 1.36x 1.5x spec_model: 1.29x - 1.62x 1.72x - 2.02x spec_plugin: 1.57x - 1.76x 2.29x - 2.37x spec_sqlite: 1.75x - 1.86x 2.26x - 2.65x spec_postgres: 1.32x - 1.40x Untested

�192




@the_zenspider

Distribute Across VMs

�193




@the_zenspider

Speed Up CI

�194




@the_zenspider

Config? Setup?

�195




@the_zenspider

Minitest will expect:SSH accessOpen ports (or tunneling?)Same source layout

�196




@the_zenspider

Safe to Assume?

�197




@the_zenspider

Distribution / Parallelism libraries:

tmm1/test-queuejeremyevans/minitest-parallel_forkArturT/knapsack

�198




@the_zenspider

Serialization Guarantees

�199




@the_zenspider

No Marshal Code

�200




@the_zenspider

Added Result

�201




@the_zenspider

Disconnected from Test

�202




@the_zenspider

Better Parallel & Distributed Testing

�203




@the_zenspider

Better GC

�204




@the_zenspider

Ruby 2.5 Exceptions?

�205




@the_zenspider

CleanerProject

�206




@the_zenspider

Split Files

�207




@the_zenspider

Cleaner Call Stack

�208




@the_zenspider

Not Sure

�209




@the_zenspider

Current StackMinitest.autorun Minitest.run(args) Minitest.__run(reporter, options) Runnable.runnables.each do |runnable| runnable.run(reporter, options) self.runnable_methods.each self.run_one_method(self, method, reporter) Minitest.run_one_method(klass, method) klass.new(runnable_method).run

�210




@the_zenspider

Minor RefactoringMinitest.autorun Minitest.run(args) Minitest.__run(reporter, options) methods = filter Runnable.runnables methods.each do |runnable| runnable.run(reporter, options) self.runnable_methods.each self.run_one_method(self, method, reporter) Minitest.run_one_method(klass, method) klass.new(runnable_method).run

�211




@the_zenspider

Dream RefactoringMinitest.autorun Minitest.run(args) Minitest.__run(reporter, options) methods = filter Runnable.runnables methods.shuffle.each do |(runnable, method)| runnable.run(reporter, options) self.runnable_methods.each self.run_one_method(self, method, reporter) Minitest.run_one_method(klass, method) klass.new(runnable_method).run

�212




@the_zenspider

Dream VersionMinitest.autorun Minitest.run(args) Minitest.__run(reporter, options) methods = filter Runnable.runnables methods.shuffle.each do |(runnable, method)| runnable.run_one_method(self, method, reporter) Minitest.run_one_method(klass, method) klass.new(runnable_method).run

�213




@the_zenspider

Too Far?

�214




@the_zenspider

Parallel By Default

�215




@the_zenspider

Powerful!

�216




@the_zenspider

Problematic

�217




@the_zenspider

DB Testing (non-transactional)

�218




@the_zenspider

Time.now�219




@the_zenspider

Code Changes

�220




@the_zenspider

command-line runner

�221




@the_zenspider

bin/minitest

�222




@the_zenspider

Better Integration eg: s/minitest/minitest_bisect/

�223




@the_zenspider

Rake’s TestTask

�224




@the_zenspider

minitest/test.rake

�225




@the_zenspider

Improved Mocks/Stubs

�226




@the_zenspider

stub w/ blocks

�227




@the_zenspider

Adding fake?obj = fake a:42

assert_equal 42, obj.a

�228




@the_zenspider

Too Simple?def fake attribs obj = Object.new attribs.each do |name, value| obj.define_singleton_method name do value end end obj end

�229




@the_zenspider

Stricter and Better Assertions.

�230




@the_zenspider

assert_equal(nil, x) => Failure

�231




@the_zenspider

More Explicit Signals Intent

�232




@the_zenspider

Fix Tautological

Tests�233

http://randycoulman.com/blog/2016/12/20/tautological-tests/




@the_zenspider

Assertion Reuse

�234




@the_zenspider

assert_operator assert_predicate

call assert_respond_to

�235




@the_zenspider

Better FailuresFailure: Expected X to respond to message.

instead of:

Error: NoMethodError (undefined method `message’ for #<X:Y>).

�236




@the_zenspider

Better Failure Messages

�237




@the_zenspider

Especially w/ layered assertions

�238




@the_zenspider

Removed assert_send

�239




@the_zenspider

Improved Specs

�240




@the_zenspider

No more expectations on Object- class Object # :nodoc: - include Minitest::Expectations unless ENV["MT_NO_EXPECTATIONS"] - end

�241




@the_zenspider

Easy Conversion# beforeo.m.must_equal 42

# after_(o.m).must_equal 42

�242




@the_zenspider


# after_(o.m).must_equal 42value(o.m).must_equal 42 # or

�242




@the_zenspider


# after_(o.m).must_equal 42value(o.m).must_equal 42 # orexpect(o.m).must_equal 42 # ugh

�242




@the_zenspider

Much Cleaner

�243




@the_zenspider

Closing

�244




@the_zenspider

Introduction History

Philosophy�245




@the_zenspider

Major Changes

�246




@the_zenspider

Architectural

�247




@the_zenspider

Parallelism

�248




@the_zenspider

Distributed Testing &

Serialization�249




@the_zenspider

Call Stack

�250




@the_zenspider

Code Changes

�251




@the_zenspider

bin/minitest

�252




@the_zenspider

minitest/test.rake

�253




@the_zenspider

minitest/mock

�254




@the_zenspider

Improved Assertions

�255




@the_zenspider

Cleaner Specs

�256







Incre

dibly

Fast

Tests


01

2

3

45

6 78

9

1011



Benchmark Tests


tor A

sser

tions


Distributed Testing





https://xkcd.com/670/




@the_zenspider

That’s It!

�259




@the_zenspider

Schedule?

�260




@the_zenspider

Schedule?Unknown!

�260




@the_zenspider

Schedule?Unknown!

Soon?!?�260




@the_zenspider

Weeks to months

�261




@the_zenspider

Not Longer (I hope?)

�262




@the_zenspider

I could use feedback and help!

�263




@the_zenspider

Current draft is available at https://github.com/zenspider/minitest

on the mt6 branch.�264

https://github.com/zenspider/minitest




@the_zenspider

Thank You




@the_zenspider

Thank YouIf you want to sponsor my work on Open Source:




@the_zenspider

Thank YouIf you want to sponsor my work on Open Source:

https://www.patreon.com/zenspider

Documents

Minitest 6: Test Feistier! Ryan Davis, Seattle.rb Minitest 6 · Presentation Sammich 7. Minitest 6: Test Feistier! RailsConf 2018, Pittsburgh, PA Ryan Davis, Seattle.rb