Estimating Costs Of Unit Testing With Qt/C++

The Background

Last summer I joined a development team where a Qt/C++ thin client application is being developed. During the early phase of the project the developers were coding a lot of fundamental base classes. However, in order to actually see something, features followed soon at the cost of unit tests for those base classes. In December we decided to allocate some effort in pushing up the test coverage that was neglected before.

We had about two weeks time and assigned each of us a module. I ended up with module X and the lead architect asked me for an estimate. So I spend some hours analyzing what’s already tested, what’s uncovered, and delivered a PERT estimate.

There’s a really great explanation of PERT by a former co-worker of mine, Michael Jendryschik, however, it is in German language: “Spiel mit den Wahrscheinlichkeiten”. For English readers I believe these blogs are a good starting point, too: “The Value of PERT Estimating” and “Basic PERT Estimate Tutorial”.

Classifying Methods And Code Blocks

From the unit tests I did before on other classes with Qt’s QTestLib I decided to classify the methods to be tested into four classes:

  • Class 1: A method containing a simple code block of no more than 20 lines of code.
  • Class 2: A method containing two code blocks of no more than 20 lines each, e.g. an if-else-statement.
  • Class 3: A method with a switch-statement containing at least 3 case blocks of about 20 lines of code each.
  • Class 4: A method with more than 20 lines of code, many nested if-statements, multiple switch-statements, and/or numerous case blocks.

Based on my experience I can give a rough minimum/average/maximum estimate for these classifications without actually looking at any line of code. Those numbers are required to do a PERT estimate. The following table shows for each class my min/avg/max estimates in [hours], the standard deviation and the calculated times with 50%, 84%, and 98% probability of occurrence. Those calculated times will be used as factors later.

PERT Estimated Hours Per Classsified Code Block
Min Avg Max σ 50% 84% 98%
Class 1 0.50 0.75 3.00 0.42 1.08 1.50 1.92
Class 2 1.00 2.00 8.00 1.17 2.83 4.00 5.17
Class 3 3.00 4.00 24.00 3.50 7.17 10.67 14.17
Class 4 8.00 16.00 40.00 5.33 18.67 24.00 29.33

Analyzing The Situation

Having a classification scheme and my estimate factors I started to look at the untested methods in the classes of module X. I correlated them to my four classes and the following table shows what I found:

Image: © Norton Russell – Fotolia.com

Source Code Analysis With Classsified Code Blocks
Source File Covered % Line Coverage Class 1 Class 2 Class 3 Class 4
AMMC.cpp 63% 58/91 1
AME.cpp 97% 33/34 1
AAM.cpp 0% 0/23 8
ARM.cpp 0% 0/39 9 1
ATM.cpp 60% 82/136 11 1
CTM.cpp 61% 36/59 3
ADVC.cpp 16% 76/472 7 4 2 2
ALVC.cpp 79% 94/119 2 1
AMKV.cpp 87% 162/185 3
AMC.cpp 48% 405/834 10 3 1
ASVC.cpp 28% 26/92 7 1
NAUDVC.cpp 13% 17/132 1 3
OUDVC.cpp 13% 17/132 1 3
43% 1006/2348 64 16 3 3

In order to extend test coverage on all missing methods I’d have to write unit tests covering 64 class 1 methods, 16 class 2 methods, and 3 class 3 and class 4 methods each. Multiplying those numbers with the factors from the first table gives me the PERT estimates that every professional software developer is expected to communicate. Look at the established truth of the numbers:

PERT Estimate With Probabilities
50% 84% 98%
Class 1 69.3 h 96.0 h 122.7 h
Class 2 45.3 h 64.0 h 82.7 h
Class 3 21.5 h 32.0 h 42.5 h
Class 4 56.0 h 72.0 h 88.0 h
Totals in PDs 24 33 42

The numbers show that covering every remaining method of module X would take at least 24 project days (1 PD = 8 hours) but chances are only at 50% that the work is actually done after 24 PDs. Adding the standard deviation gives you a probability of 84% but that would take probably 33 PDs. After 42 PDs (50% value plus twice the standard deviation) there’s a pretty high chance that all methods will be covered by unit tests but be careful, it still ain’t a 100% exact estimate.

How It Compares To Reality

To improve coverage of module X I had 12 PDs. If you look at the PERT estimate above it would take at least 24 PDs at a 50% chance to cover all remaining methods.
For the last 6 days a co-worker joined in on my module, so in total 18 PDs were spend on writing unit tests for module X in 12 days.
In case you think we made it in 18 PDs: wake up and think again. Having done those PERT estimates I knew noone would and should expect anything close to 90 or 80 percent coverage after 18 PDs. And that’s exactly reflecting the reality:

Before After 18 PDs Coverage Comparison
Source File Covered % Line Coverage Covered % Line Coverage
AMMC.cpp 63% 58/91 100% 97/97
AME.cpp 97% 33/34 100% 35/35
AAM.cpp 0% 0/23 - -
ARM.cpp 0% 0/39 - -
ATM.cpp 60% 82/136 99% 137/138
CTM.cpp 61% 36/59 98% 58/59
ADVC.cpp 16% 76/472 80% 335/418
ALVC.cpp 79% 94/119 83% 99/119
AMKV.cpp 87% 162/185 91% 159/175
AMC.cpp 48% 405/834 55% 466/842
ASVC.cpp 28% 26/92 63% 58/92
NAUDVC.cpp 13% 17/132 91% 125/137
OUDVC.cpp 13% 17/132 92% 127/138
43% 1006/2348 75% 1696/2250

Within 18 PDs the total coverage in module X was improved from 43% to 75%. If you look at the line coverage you’ll find that due to the tests and the code review along with it some classes lost or gained lines. We even got rid of two classes completely.
So, we had a coverage of 43% before writing the unit tests. 57% was the way to go and most likely that would have taken about 38 days. With those 18 PDs we spend we made 32% and that’s roughly half of the way in about half of time that was required.

Of course, I can give no guarantee nor warranty for my classification and factors to work on your project, too. Also, there was not enough time, as usual, to achieve full coverage. So even the 32% we made in 18 PDs might raise a wrong expectation that 100% coverage would’ve been achieved after about 38 PDs. But still, the 84% PERT estimates for each single CPP file (not shown in this blog post), matched pretty well.

© DoraZett - Fotolia.com

What these numbers show also is: Unit testing is expensive! Covering the missing 1342 lines of code would cost very likely about 38 PDs. It is no wonder that many managers and customers don’t want to pay for testing. That’s also why developers should carefully spend their effort in it. Don’t over test. Deliver what’s promised in contract but no more. Customers usually pay for what they see and feel: UI and usability. Although I am a professional software developer I make no bones about disliking unit tests. I do them only, if they are explicitly payed for. The only really relevant tests are functional and acceptence tests as they proove that your software does what it’s payed for. And if those pass, usually anything more low level works the way expected.

One More Thing

If you are a developer and not familiar with PERT estimates yet, please, start getting familiar with it. Start doing PERT estimates for yourself. You’ll gain more confidence when you learn about what you can do how fast. When communicating your PERT numbers never forget to quote the probability to your superior. Following that little guideline will lift a lot of weight from your shoulders and it will also help your superior to do his math better. It’s an insurance for both sides.

Thanks for reading.

Request an offer for an inhouse workshop and coaching about PERT.

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>