eigenclass logo
MAIN  Index  Search  Changes  PageRank  Login

Better code coverage for Ruby: rcov 0.1.0 prerelease

Update^2: newer version available
Update: explanation of C0-C2 code coverage analysis

rcov is a code coverage tool for Ruby I originally wrote 2 years ago. It improves on the better known coverage in a number of points:

  • 20-300 times faster
  • more accurate coverage information through code linkage inference using simple heuristics
  • slightly more convenient interface (arguably; work on this is probably needed)
  • additional features like bogo-profile mode, colorblind-safeness...

Code coverage shouldn't be abused (in few words, C0 coverage guarantees nothing) but it's still useful for testing: it will at least tell you when your tests need more work, and most importantly where.

What does it look like?

Just use rcov to run your program (instead of Ruby), and a number of XHTML files with the code coverage information will be generated.

For example, given

srand(0)
c = 0
d = 1
10.times do |i|
  if rand % (i + 10)
    c += i
    d += 1
  end
end
puts "blergh"
if c > 4*d
  # stuff
  puts "yep"
else
  puts "nope"
  # more stuff
end

puts "Done: #{c+d}"

you'll get something resembling /hiki/screenshot.png

'"Code coverage" what??': an explanation of C0, C1 & C2 coverage analysis

C0 coverage analysis (as done by rcov) tells you which lines of code have been executed: you typically use it to find the areas of your program that have not been sufficiently tested, i.e. those that were not run by your test cases. C1 coverage refers to the different branches of conditional statements, for instance,

if somecondition; dofoo; else dobar end

gets 100% C0 coverage (you did "execute that line") , but only 50% C1 coverage if only one branch was followed: iow. in order to get 100% C1 coverage you'd have to change your tests so that somecondition is true in a run and false in another.

C2 (path) coverage corresponds to the amount of execution paths through your code that actually happened:

if foo == 1
  do1()
else
  do2()
end

if bar
  do3()
else
  do4()
end

The possible execution paths are:

  • do1 do3
  • do1 do4
  • do2 do3
  • do2 do4

In order to get a C2 coverage of 100%, you'd have to ensure that all those combinations (cartesian product of {foo == 1, foo != 1} and {bar, !bar}) happen.

It's possible to get very close to 100% C0 and C1 coverage. While a C0-1 coverage of 100% doesn't guarantee that your program is correct, if your coverage is much lower it certainly means that your tests aren't good enough. As for C2 coverage, it's usually not possible to make it very high: you can keep it as an indirect measure of quality, but the gain/cost ratio is lower than for C0-1 coverage, I guess.


Download

rcov-0.1.0.tar.gz

Just run

 ruby setup.rb

to install it. If you cannot compile extensions for some reason, follow the instructions in README.en.

Why pre-release now?

I'm pre-releasing rcov 0.1.0 in order to force myself to release it for real eventually. It's a simple tool that works basically right for me, but I'm willing to polish it some more for a release. The more feedback I get, the sooner that will happen*1. The heuristics, for instance, could be improved considerably, and the interface could use some work.


Wanted feedback? here it comes - riffraff (01-02-2006 (Mer) 14:00:43)

It seem that rcov gets confused by multiline arguments that include multiline expressione themselves, such as foo(

[
 :x, 
 :y
]

) or foo(

foo(
 :x, 
 :y
)

)

I'm sorry I can't provide a patch, but rcov code is quite dense and missing tests, and I'm not a great hacker..


mfp 2006-02-24 (Fri) 13:27:16

I don't really believe you when you say you're not a great hacker, but the code is indeed ugly *g* I'll try to write some unit tests before expanding the heuristics.

UI: colour key? - hgs (2006-01-24 (Tue) 03:50:10)

This will definitely be a useful tool to have. Thank you.

It would seem useful to add a key explaining what the different colours mean in the HTML output. If it is there, I don't see it.

I'm wondering if it would be practical to yield the source lines to a block so the text can be syntax highlighted (by something else)? It could result in nested spans or something...

A purely textual output could be useful as well, on the basis that it would be simpler for parsing by another program. I seem to recall that ZenTest can generate tests for a program automatically, so someone may wish to do something similar given coverage information.


mfp 2006-01-24 (Tue) 05:12:25

In the above image, lines in gray were not executed; those with a light green background were reported as run by ruby, and those in darker green shades were inferred by rcov as run. The latter is necessary because ruby only reports 1 usable event in multi-line statements (and of course ignores blank/comment lines altogether). I'm adding your ideas to my mental TODO, thank you :) I've been thinking about something along the last one, I'll try to explore that.

I wonder... - riffraff (23-01-2006 (Lun) 15:48:17)

how would it fare with a behemoth like rails :) Anyway thanks for sharing this (again)


mfp 2006-01-24 (Tue) 05:05:52

According to http://alexpooley.com/articles/2006/01/01/ruby-on-rails-test-coverage, some ppl found the older version of rcov and actually used it to assess functional & unit tests in Rails applications. The present version is two orders of magnitude faster, so the tests will usually run only ~2-5 times slower than normal, instead of 40-1500...


riffraff 31-01-2006 (Mar) 15:09:25

just tried it.. works like a charm with rails, even if it would be convenient if it accepted multiple files in input, since rails apps tipically don't have a test_all.rb that groups all the test files

Forgive the ignorant... - Matt Todd (2006-01-23 (Mon) 11:55:17)

I'm sorry, you must forgive my ignorance and lack of experience, but what exactly is the meaning of this usage of the term 'coverage'?

M.T.


mfp 2006-01-24 (Tue) 05:00:21

Just added a short explanation of C0-2 code coverage analyses, HTH. Thanks for reminding me that my description of rcov was incomplete (^_^)

Last modified:2006/01/24 05:15:14
Keyword(s):[blog] [ruby] [rcov] [code] [coverage] [testing] [release]
References:

*1 It takes some stimulus to overcome my excessive laziness regarding releasing. I'm not joking; this pre-release was triggered by a request for info on rcov from Michael Burrows, so if you first heard about rcov here and you like it at all, thank him for that.