Which code coverage tool to choose?

We make major investments in tools, people and process to effectively test applications. But how we measure the effectiveness of these investments? How do we know how well our applications are getting tested? How do you know what percentage of application is getting tested? There is a famous quote: “If you can’t measure it, you can’t improve it”, it’s very well applicable to testing as well.

Answer to the above question is test coverage tools also popularly known as code coverage tools. Now the next question becomes:  ‘Which code coverage tool to choose?’

We also stumbled on the same question. As all of our applications are running on JVM, we evaluated major java based code coverage tools:

• Clover
• Cobertura
• JaCoCo

This document intends to walkthrough our selection process.

Licensing Model

Licensing model is an important criteria as it translates to real $$. Clover is a commercial product from atlassian, who is known for developing great software products such as: JIRA, Confluence.

Both Cobertura and JacCoCo are open source products.

Google Trend

We don’t want to pick a product whose adoption is on a downward trend. How can you judge the adoption rate? One handy tool to answer this question is Google Trend.

Fig: Code coverage tools trend over last 10 years

From the google trend we inferred that definitely Clover is declining and underperforming over Cobertura and JaCoCo. On the other hand there is a tough competition going between Cobertura and JaCoCo with JaCoCo having a minor lead in recent times.

GitHub Star rating

Both Cobertura and JaCoCo are hosted in the popular GitHub SCM. For each project they have star rating. Star Rating is a mechanism to indicate how many people are keeping track of the repositories. It’s another good way to see the popularity of the tool.

Fig: Cobertura’s star rating in GitHub

Fig: JaCoCo’s star rating in GitHub

You can notice that Cobertura has 312 stars, whereas JaCoCo has 348 stars. Start rating is more or less similar, with JaCoCo having a small edge over Cobertura. This pattern matches with Google trending report

Since Clover is a commercial product, it’s not publically hosted on GitHub, does there is no star rating associated to it.

Instrumentation Methodology

Both Clover & Cobertura supports only runtime instrumentation, whereas JaCoCo supports both compile time and runtime instrumentation. I truly believe providing compile time instrumentation on top of run time instrumentation is a major advantage. One downside with run time instrumentation is: you can get coverage report only until the point JVM is running. Once it’s restarted and if you run additional tests, you will be able to get only new coverage report and not the cumulative report of the tests that were conducted before the restart. In large enterprises there would be unit tests, functional tests, regression tests, smoke tests…. each would be owned by different teams. So if I want to get a cumulative code coverage report, compile time instrumentation is the easier way to achieve it. I am not saying that with just runtime instrumentation you cannot achieve, I am just saying it would be hard to achieve.

Ease of setup

This is just personnel experience, it can’t be generalized to everyone. But personally we found setting up Cobertura and JaCoCo to be quite trivial when compared to clover. We were able to accomplish the setup in couple of hours, whereas Clover was tougher to setup, we ran in to few issues

Reporting

Without a comparison Clover’s reporting/dashboarding capability of Clover was way ahead then both Cobertura and JaCoCo. Visual representation, Graphical representation, coverage map, project metrics were all quite superior then other two products. Below goes the index page of the report of each tool..

Fig: Clover report

Fig: Cobertura Report

Fig: JaCoCo report

IDE Integration & other features

Of all the 3 tools, JaCoCo has the best integration with Eclipse IDE. It’s Runtime adds a coverage mode button right next to Run and Debug buttons. By clicking this button you can launch your applications or unit tests in coverage mode to collect coverage statistics. It’s very trivial and easy to use.

Similarly another great feature in JaCoCo is Multiple Session Coverage. Typically as I mentioned in the ‘Instrumentation methodology’ section large enterprise organizations will have multiple tests/test suites: unit tests, functional tests, regression tests, smoke tests… each would be owned by different teams. So if you want to get one unified view of all tests coverage this multiple session coverage feature becomes invaluable.

Summary

Below table summarizes entire analysis process:

	Clover	Cobertura	JaCoCo
Licensing Model	Commercial	Open Source	Open Source
Google Trend	Downward	Rising	Rising
Github star rating	n/a	358 stars	415 stars
Instrumentation	Runtime	Runtime	Runtime & Compile time
Ease of setup	Comparatively difficult	Easy	Easy
Reporting	Best	OK	OK
Eclipse IDE Integration	None	None	Good. Eclemma Plugin.
Multiple Session Coverage	None	None	YES

As we entertain open source solution and wanted the multiple session coverage and easy integration with Eclipse, we decided to use JaCoCo. It doesn’t mean other tools are inferior to JaCoCo. Each tool has its own strength, we decided to choose JaCoCo has it suites our need more.