Sonarqube With Opencover And xUnit Reports

Previously Ive used Sonarqube to analyze without test coverage, recently Ive learnt how to use opencover to generate code coverage stats and use XunitXml.TestLogger to generate the test coverage report, this data is then used by Sonarqube to generate reports on the code base.

Software Setup

Sonarqube Server

  1. Start the container, the default login is admin\admin, here Im using v8.3 community
1
docker run -d --name sonarqube83c -p 9000:9000 sonarqube:8.3-community

Sonar Scanner

  1. Install the dotnet-sonarscanner tool into the path c:\dev\sonardemo\tools
1
dotnet tool install dotnet-sonarscanner --tool-path tools --version 5.0.4

.Net Core SDK

The source code Im going scan is netcoreapp2.2, I think later versions will be backwards compatible so you could skip this step.

  1. Download and install SDK 2.2.207

On my machine this installs to C:\Program Files\dotnet\sdk\2.2.207

Setup Source Code

Clone And Build

  1. Clone the project VulnusCloud to c:\dev\sonardemo\tmp, I picked this project for a few reasons:
  • it uses NUnit and I wanted to see if this would work the same as it does for a project using xUnit, turn out it does. FYI xUnit is more popular and its the default at most companys Ive worked for.
  • its got muiltiple test projects, I want to only run the UnitTests and ignore the IntegrationTest in the coverage report, currently the sln only references the UnitTests, I’ll include the IntegrationTest to test excluding them from the dotnet build and test steps.

  1. Copy the contents of C:\dev\sonardemo\tmp\VulnusCloud to C:\dev\sonardemo\ so that the .sln file is in the root, this just makes the sonar steps easier.

  2. Check the solution builds

1
dotnet "C:\Program Files\dotnet\sdk\2.2.207\MSBuild.dll" .

Add Dependencies

  1. We need to install XunitXml.TestLogger
1
2
C:\dev\sonardemo\UnitTests
dotnet add package XunitXml.TestLogger --version 2.1.26
  1. Additionally we need a collector, the de facto is coverlet.collector. coverlet.collector is a tool specifically designed to measure code coverage for .NET applications running on various platforms
1
2
C:\dev\sonardemo\UnitTests
dotnet add package coverlet.collector --version 1.3.0
  1. Create a .runsettings file in C:\dev\sonardemo\UnitTests, the friendlyName="XPlat code coverage" refers to the measurement of how much of your codebase is executed during testing across different platforms (hence, “cross-platform” or “XPlat”)
1
2
3
4
5
6
7
8
9
10
11
12
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>opencover</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Other examples from the docs include

1
2
3
4
...
<Configuration>
  <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
  <ExcludeByFile>**/dir1/class1.cs,**/dir2/*.cs,**/dir3/**/*.cs,</ExcludeByFile>

Runsettings can be added in UnitTests.csproj with in the PropertyGroup, the downside is then when you run the tests from an IDE like Visual Studio it will create TestResults folder in the root, adding to the csproj file is not required because we will pass it with the dotnet test command so I just mention it here for completeness.

1
2
3
4
<PropertyGroup>
  ...
  <RunSettingsFilePath>$(MSBuildProjectDirectory)\.runsettings</RunSettingsFilePath>
</PropertyGroup>

Also see

Oh Its Scan time! 😬

You can use sonar.login and pass a key like I did here but passwords belong in source code right? (I do what I want 🙈)

  1. Run scanner begin
1
2
3
4
5
6
7
./tools/dotnet-sonarscanner.exe begin `
    -d:sonar.login=admin `
    -d:sonar.password=admin `
    -d:sonar.host.url=http://localhost:9000 `
    -k:VulnusCloud `
    -d:sonar.cs.opencover.reportsPaths='UnitTests\TestResults\**\coverage.opencover.xml' `
    -d:sonar.cs.xunit.reportsPaths='UnitTests\TestResults\xunit.report.xml'
  1. Run dotnet tests, if you dont specify the --framework argument it will use what ever version of dotnet thats in your systems PATH environmental variables, here Im rolling with .Net Core 2.2
1
2
3
4
dotnet test VulnusCloud.sln `
    --settings './UnitTests/.runsettings' `
    --logger 'xunit;LogFilePath=TestResults\xunit.report.xml' `
    --framework netcoreapp2.2

Along with running the tests it creates the XML reports listed below, you could delete them as a run step, however sonarscanner keeps track of the based on the path but if this is running in a CI/CD pipeline on a volatile TeamCity agent the report file probably wont exist on the next run. Ta ta ma chance, uhambe kahle mfowethu ❤️

  • C:\dev\sonardemo\UnitTests\TestResults\20cbc40e-1bb1-449e-a3e2-3d1a33c75315\coverage.opencover.xml
  • C:\dev\sonardemo\UnitTests\TestResults\xunit.report.xml
  1. Run scanner end
1
2
3
./tools/dotnet-sonarscanner.exe end `
    -d:sonar.login=admin `
    -d:sonar.password=admin

The code analysis can then be seen at http://localhost:9000/dashboard?id=VulnusCloud

Yeah Boi

xUnit Report Path Defaults

Adding sonar.cs.xunit.reportsPaths is not actually required, if you omit it the report will be added as TestResults.xml and picked up automagically, the resulting XML report paths are the same as the above. LogFilePath is then also not needed for the dotnet test logger argument (is that an arguments, argument? LOL)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
./tools/dotnet-sonarscanner.exe begin `
    -d:sonar.login=admin `
    -d:sonar.password=admin `
    -d:sonar.host.url=http://localhost:9000 `
    -k:VulnusCloud2 `
    -d:sonar.cs.opencover.reportsPaths='UnitTests\TestResults\**\coverage.opencover.xml'

dotnet test VulnusCloud.sln `
    --settings './UnitTests/.runsettings' `
    --logger xunit `
    --framework netcoreapp2.2

./tools/dotnet-sonarscanner.exe end `
    -d:sonar.login=admin `
    -d:sonar.password=admin

File Level Exclusions

  1. We need to install coverlet.msbuild for this to work
1
2
C:\dev\sonardemo\UnitTests
dotnet add package coverlet.msbuild --version 2.9.0
  1. Then add annotation [ExcludeFromCodeCoverage] //Justification I do what I want with valid justification. I’ve seen //NOSONAR comments in some code bases but couldnt get it to work, maybe oneday I’ll figure it out and add it here.

ExcludeFromCodeCoverage examples

1
2
3
4
5
6
namespace Business
{
    [ExcludeFromCodeCoverage] // class level
    public class OssReportService : IOssReportService
    {
        private readonly IReportRepository _reportRepository;
1
2
3
4
5
6
...
    [ExcludeFromCodeCoverage] // method level
    public async Task ProcessOssRecords(DateTime dateTimeOfMethodCall) 
    {
      ...
    }

We can then see the coverage value increases:

Coverage goes up

Additionally the file level exclusions no longer have the red line on the left:

File level exclusion

Parameter Level Exclusions

See Analysis scope to understand more about pattern matching used below.

sonar.exclusions

Configure the files that should be completely ignored by the analysis, so this is things like bad code and code smells.

1
-d:sonar.exclusions="**/Startup.cs,**/Program.cs,**/IoC/**,**/Dtos/**,**/Constants/**,**/Models/*"

sonar.coverage.exclusions

Configure the files that should be ignored by code coverage calculations, so this is just for test coverage.

1
-d:sonar.coverage.exclusions="**/SomeCrappyServiceThatIProbablyWontFixLater.cs,**/FooService/*"