Turbo Charged Test Suites

Turbo Charged Test Suites

Tuesday, 7 October 2008

Curtis “Ovid” Poe

• CPAN Author

• “Perl Hacks” Contributor

• Perl Foundation Member


Test Suite Performance

• Computer

• Developer


Faster Tests


Considerations

• Why speed things up?

• Trade-offs


Before You Start

• Set a goal

• No tests may fail

• No non-test output


Databases

• Don’t drop them

• Don’t fake it with transactions


Databases

• Static tables -- country_codes

• Dynamic tables -- orders


CREATE TABLE changed_table ( changed_table_id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, table_name VARCHAR(30) NOT NULL, is_static INT NOT NULL DEFAULT 0, inserts INT NOT NULL DEFAULT 0, updates INT NOT NULL DEFAULT 0, deletes INT NOT NULL DEFAULT 0)

-- then insert table records


foreach my $action (qw/insert update delete/) { $dbh->do(<<" END_SQL"); CREATE TRIGGER tr_${action}_$table BEFORE $action ON $table FOR EACH ROW UPDATE changed_table SET ${action}s = ${action}s + 1 WHERE table_name = '$table'; END_SQL}


db.disable_foreign_keys()for table in changed_tables() table.truncate() if table.is_static() table.rebuild()db.enable_foreign_keys()


Databases

• Before: 80 minutes

• After: 22 minutes


Aggregation

• Only load things once

• Dangers of shared state


use My::YAML::Test;My::YAML::Test->run;# t/api/customer.t# t/api/customer.yml


use My::YAML::Test;foreach (yaml_tests()) { diag “running $_”; My::YAML::Test->run($_);}

sub yaml_tests { return @ARGV if @ARGV; # or return all YAML files ...}


YAML Aggregation




use Test::Aggregate;my $tests = Test::Aggregate->new({ dirs => ‘aggtests’,});$tests->run;


use Test::Aggregate;my $tests = Test::Aggregate->new({ dirs => 'aggtests', dump => $dump, set_filenames => 1, shuffle => 1, startup => \&startup, shutdown => \&shutdown,});


Generic Aggregation




Better Aggregation

• use Test::Class


OK to Fail is OK

• POD tests

• Perl::Critic

• Anything “non-functional”


OK to Fail is OK

• Move them to xt/

• But they must be run!


Lessons Learned

• Not dropping your database is tricky

• Aggregating tests finds bugs in tests

• Aggregating tests finds bugs in code

• *CORE::GLOBAL:: is evil

• Don’t touch UNIVERSAL::


Faster Programmers


Custom Test Modules


use Test::More tests => 13;use Test::Exception;use Test::XML;use Test::JSON;use Test::Differences;


package Our::Test::More;

use Test::Builder::Module;our ( @ISA, @EXPORT );use Test::More;use Test::Exception;

BEGIN { @ISA = qw(Test::Builder::Module); @EXPORT = ( @Test::More::EXPORT, @Test::Exception::EXPORT, );}

1;


package My::Custom::Tests;

use Test::Kit ( 'Test::More', 'Test::XML', 'Test::Differences', '+explain',);

1;


Popularity ContestTest::More 44461Test 8937Test::Exception 1397Test::Simple 731Test::Base 316Test::Builder::Tester 193Test::NoWarnings 174Test::Differences 146Test::MockObject 139Test::Deep 127


use Test::Most tests => 4, 'die';

ok 1, 'one is true';is 2, 2, '... and two is two';eq_or_diff [3], [4], “... but three ain’t four”;throws_ok { $foo/0 } qr/Illegal division by zero/, 'and no-constant folding with vars';


Test From Your Editor


“ in your .vimrc

function! PerlMappings() noremap K :!perldoc <cword> <bar><bar> \ perldoc -f <cword><cr>endfunction

function! PerlTestMappings() noremap <buffer> ,t :!prove -vl %<CR>endfunction

“ remember My::Test::YAML?function! YAMLTestMappings() noremap <buffer> ,t :!prove -vl t/yaml.t :: %<CR>endfunction

au! FileType perl :call PerlMappings()au! FileType yaml :call YAMLTestMappings()au! BufRead,BufNewFile *.t :call PerlTestMappings()


vim $(ack -l --perl 'api/v1/episode' t/)

map <leader>tb :call RunTestsInBuffers()<cr>function! RunTestsInBuffers() let i = 1 let tests = '' while (i <= bufnr("$")) let filename = bufname(i) if match(filename, '\.t$') > -1 let tests = tests . ' "' . filename . '"' endif let i = i+1 endwhile if !strlen(tests) echo "No tests found in buffers" else execute ':!prove ' . tests endifendfunction


Advanced “prove”


Test-Harness $ prove -l t --state=hot,fast,save --timer[20:57:19] t/yamlish-output.........ok 40 ms[20:57:20] t/console................ok 45 ms[20:57:20] t/utils..................ok 48 ms<snip>[20:57:23] t/harness................ok 300 ms[20:57:23] t/process................ok 1020 ms[20:57:24] t/prove..................ok 1017 ms[20:57:25] t/regression.............ok 4217 ms[20:57:29]All tests successful.Files=32, Tests=10326, 10 wallclock secs ( 1.25 usr 0.24 sys + 5.60 cusr 1.83 csys = 8.92 CPU)Result: PASS


Test-Harness $ prove -l t --state=hot,slow,save -j 9 t/callbacks.............. ok t/nofork................. ok t/proverc................ ok <snip>t/yamlish................ ok t/prove.................. ok t/regression............. ok All tests successful.Files=32, Tests=10326, 6 wallclock secs ( 1.34 usr 0.24 sys + 5.63 cusr 1.83 csys = 9.04 CPU)Result: PASS


# slow running tests from App::Prove::State# http://use.perl.org/~Ovid/journal/35831

Generation 18Number of test programs: 58Total runtime approximately 17 minutes 35 secondsFive slowest tests: 482.7 seconds -> t/acceptance.t 234.4 seconds -> t/aggregate.t 96.3 seconds -> t/standards/strict.t 66.6 seconds -> t/unit/db/migrations.t 56.7 seconds -> t/unit/piptest/pprove/testdb.t


http://use.perl.org/~Ovid/journal/35831

http://use.perl.org/~Ovid/journal/35831

Devel::CoverX::Covered


• What tests cover this file?

• What files are covered by this test?

• (Soon) What tests cover this line?

• (Soon) What tests cover this subroutine?

• ... and more ...

Devel::CoverX::Covered


function! PerlMappings() noremap <buffer> ,cv :call Coverage()<cr>endfunction

function! PerlTestMappings() noremap <buffer> ,t :!prove -vl --norc %<CR>endfunction

function! Coverage() let file = bufname('%') if match(filename, '\.t$') > -1 execute '!covered by --test_file="'.file.'"' else execute '!covered covering --source_file="'.file.'"' endendfunction

au! FileType perl :call PerlMappings()au! BufRead,BufNewFile *.t :call PerlTestMappings()


Questions?


Technology

Turbo Charged Test Suites