UnitTestMonitor: Monitoring Qt unit tests

UnitTestMonitor showing testresults over time

It’s been bothering me ever since that there’s no integration for unit tests in QtCreator. As I’m quite spoiled by the JUnit support in Eclipse IDE. Last summer I started to work on a tool that I call UnitTestMonitor. It’s not integrated into QtCreator yet, still I just released version 1.0.0 on github.

I posted about “Generating LCOV Coverage with QtCreator” already. That is pretty nice as it opens the coverage report in a browser right after running the unit test from within QtCreator. But when you have a project consisting of many sub-projects with several libraries there was just no way to access all the test and coverage data from a single view.

Introducing UnitTestMonitor

UnitTestMonitor is an application assisting in monitoring unit tests and lcov coverage reports.

The application is designed to be able to analyze Qt testlib unit test output. It identifies testlib projects by name: test projects need to have the same name as the tested project but end with “Test”, e.g. if you have a project “libFoo” UnitTestMonitor will look for “libFooTest” and if found add “libFoo” to a monitor set.

Mapping of project structure to monitor set

How does it work?

In version 1.0.0 UTM is quite inflexible. It needs a fixed scheme for file names, test, and coverage report locations. A picture is worth more than words in that case. Please look at the project structure of Rowlo Persistence Framework (RPF).

The master branch of RPF is checked out into a folder "RPF" that’s located in the parent folder "RPF_master". When you add a branch to a monitor set it is that "RPF" folder that you select. UTM will then scan the contents of that folder for projects. The projects are expected to be sub-folders of the branch folder. In case of the screenshot "libRowloPersistence" and "libRowloSqlitePersistence" are project candidates.

A project candidate is accepted as project if it contains at least one library. (The main application project is a library, too, for simplicity of writing.) For UTM a library requires two sub-folders to be present in the project folder. Both must start with the same name and one of them has to end with "Test". The blue number 1 in the screenshot shows the detected "libRowloPersistence" library.

The orange lines show how UTM maps the folder structure to the table. The two top level folders are used to label the branch tab. UTM supports monitoring several branches or projects next to each other which comes in quite nice when your product has already multiple versions or is made up of several standalone components. The project and libraries are shown as rows in the tree table.

The value of UTM unfolds by being able to analyze Qt testlib unit test output and show the results in the tree table. UTM is looking for a file with the name “tlog” in the location indicated by the blue number 2: "_/tests/<libraryName>/tlog". The tlog file can be created manually by running the unit test executable with the arguments "-txt > tlog" or feeding the parameter line of QtCreator’s run configuration of each unit test with appropriate values. Results of the testcases and tests are stored in it and UTM will build a row for each testcase.

The blue number 3 indicates the folder that is scanned for an “index.html”. UTM assumes that if that file is present in "_/testcoverage/<libraryName>/" then it is the lcov coverage report.

To get started you need to create a monitor set file. It’s where UTM stores the analyzed data of your project structure and unit tests. Once done, and provided you project structure matches the above description, you can press the “Add branch” tool button on the right and select your project folder (the one that contains the libraries with your .pro files). The current timestamp of the scan will be used for the column label. When you do “Update branch” later then a new row with a new timestamp label is added on the left side of the table.

Features and Value

As shown on the first screenshot in that post, UTM keeps track of past unit test results. This is very useful for:

  • refactoring code: Make sure tests are green before refactoring. Refactor code. Anything turned red? Well, you still got work to do.
  • reporting unit test performance: UTM keeps track of the number of tests in each testcase. An increasing number over time reflects your unit test efforts.
  • assessing code quality: Are there flipping unit tests? Dig into it! A testcase is red for weeks? What the heck! Got it? Good. :-)

UTM does not keep track of the tests automatically. You need to push the “Update branch” tool button to scan for current tlog and test coverage reports. New projects or libraries will be added without further action.

Hovering the mouse shows popup with FAIL messages
UTM stores the tlog FAIL messages. Hovering the mouse over any red cell of testcase rows will show a tool tip with all FAIL messages. Hovering on red cells of library or project rows will not show that tooltip, as those messages can accumulate to a large number quickly. You need to expand the tree down to the testcases first.

Dialog with tlog of testcase
Double clicking a testcase row will open a dialog with the testcases tlog fraction. That’s very convenient as the tlog tends to become very long the more testcases there are. Finding the one you’re looking for is not what I like to do.
UTM does not store the full tlog file, so that dialog will always show the current tlog no matter what column you double click.

Dialog with lcov coverage report of library
Double clicking a library row opens a dialog with an html viewer. It will show the most recently generated lcov coverage report. Hyperlinks are navigable. You can go back via a context menu’s “Back” action.

What you can do

You’re free to do whatever you want with the source code and the tool. Use it. Hopefully, it will bring value to you, too. Leave me a comment with feedback if you like.

If you are into QtCreator source code, please contact me. I really want this to be part of QtCreator but could need a little help to get started.

Anyone willing to contribute to the public github repository is welcome to do so, too.

Download and Build

UnitTestMonitor is available on github.

To obtain the sources: git clone https://github.com/rowlo/UnitTestMonitor.git
To build using QtCreator: Open UnitTestMonitor.pro file and configure build for Qt 5.2.

If you want to adapt lcov report and tlog detection to your project structure then edit the source file BranchScanner.cpp at lines 99, 100, and 101. If you come up with a generic flexible solution I’d appreciate a contribution to the sources.

One more thing

The tool is probably far from perfect due to its need for a specific project structure. However, it is open source, provided for free, and I did this in my evening hours plus that project structure ain’t a bad choice anyways. So, please, keep that in mind before condemning it. :-)

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>