SciTools Checkers in SonarQube


Version of Document: V1.0
Author: Stephane Raynaud, Emenda USA (Stephane.Raynaud@Emenda.com)
Tested with Version 4.0 of SciTools Understand ( Build 867 )

1. Objectives

This document presents a basic integration of SciTools Understand CodeCheck and SonarQube. The integration requires two scripts that you must get from Emenda USA by emailing Stephane.Raynaud@Emenda.com (create_sonarqube-cxx-external-rules.pl and create_sonarqube-cxx-report-from-scitools-analyis.pl).
The integration allows you to display all the code violations found by SciTools Understand CodeCheck as SonarQube Issues and automatically update those.
Integrate Sonar Qube with Understand Code Check
You will be able to automate the process of updating the issues, by using a command line script. The script follows the following steps:
  1. Run an Understand Analysis (optional) from the command line.
  2. Run CodeCheck from the command line.
  3. Convert CodeCheck violations to SonarQube CXX Issues from the command line.
  4. The Violations will be picked up when running sonar-scanner automatically.
Interested? let's get to work!

2. Pre-Requisites

Before we start, you will need:
  • An Installation of SonarQube 5.6.3 (Later version are not supported because of incompatibilities with the Community C++ plugin)
  • C++ Community Plugin (version 0.9.6)
  • You will need admin access to the SonarQube
  • You will need to restart your SonarQube Server
Hint: Installing the C++ Plugin.
  1. Download sonar-cxx-plugin-0.9.6.jar from https://github.com/SonarOpenCommunity/sonar-cxx/releases/tag/cxx-0.9.6 ( bottom of page )
  2. Copy it into SONARQUBE_HOME/extensions/plugins
  3. Restart the SonarQube server
  4. Navigate to the Update Center (/updatecenter). The Update Center should list "C++ ( Community )" on the tab "Installed Plugins":
  5. By default, the "admin" login is admin/admin...

3. Setting up in SciTools.

In a first step, we are going to
  • create a UDB file
  • export the Checks from SciTools to create an XML file that is loaded in SonarQube CXX plugin as Rules.
We are basically teaching SonarQube about SciTools Checks.
This must be done only once.
3.1. Building Your Project SciTools Database
Before you do anything, you will need to create a udb database later used for your analysis. The only requirement is that the database must be using a named root for portability and this root must be the root of the project, i.e., the place where one can fine the "sonar-project.properties" file.
Hint: Using a Named Root for Portability in SciTools Understand
a) Open your project settings in SciTools Understand (Project->Configure Project)
(Alternatively, you can do this when you create the projects and add files to it)
Note: It is always a good idea to use named root when creating udb files. It provides better portability between developers (they don't all have their source code at the same place). Named Root lets you define an "environment variable" that defines the ROOT of your project.
b) Select Files
Select files
c) On the right of the window, click "Portability"
Select portability
d) Select "NamedRoot"
Select named roots
e) Lastly, add a "Named Root" using the "Edit Named Roots..." button on the bottom left. In my example, I define a LIBFREENECT2_ROOT for the root of my project, and it points to C:\Users\steph_000\Desktop\DEMOS\SonarQube\demo_project\libfreenect2-master.
Create named root
This is the only restriction on the udb file.
3.2. Exporting ALL the SciTools Checks
Now we need to "teach" SonarQube about the SciTools Checks. But before we can do this, we need to extract a list of checks from SciTools.
To do this, Open CodeCheck in your udb file.
  1. Select All the checks available (Yes!!! All of them!!!!) - This is not the checks you will use in your analysis. We want SonarQube to learn about all the Checks .
  2. On the Top Right Corner, Select "Export Selected Checks to Text File". Let's name this file ALL.txt.
    Select all Scitools checks
WARNING: DO NOT USE "Export Configuration to a File".
It's easy to get the two confused!!!
If you open the exported file, it should look like this:
CodeCheck Checks
SciTools' Recommended Checks\"Commented Out" Code
Description: Sections of code should not be "commented out".
SciTools' Recommended Checks\Definitions in Header Files
Description: Objects and Functions should not be defined in Header Files.
SciTools' Recommended Checks\Files too long
Description: Program files should not have more than the specified number of lines
SciTools' Recommended Checks\Floating-point expressions shall not be directly or indirectly tested for equality or inequality.
Description: (Required) Floating-point expressions shall not be directly or indirectly tested for equality or inequality.
...

4. Translating SciTools Information into something SonarQube can read.

We are now ready to use the script "create_sonarqube-cxx-external-rules.pl" provided by Emenda USA.
Hint: Use the perl installation from SciTools
SciTools comes with a perl installation, so why use anything else? The command to run a perl command is uperl. On my plaform is was located in <SCITOOLS>\bin\pc-win64.
It is a good idead to add it to your path!
Now Run the following command:
uperl create_sonarqube-cxx-external-rules.pl .\ALL.txt .\ALL.xml .\ALL.xml.key
  • ALL.txt is the file you exported earlier from SciTools
  • The ALL.xml and ALL.xml.key files that will be created.
  • ALL.xml will be imported in SonarQube later, and ALL.xml.key will be used at a later stage.
The output should be something like:
I will read definition of checks from .\ALL.txt
I will write definition of sonarqube rules to .\ALL.xml
I processed 640 rules
If you open ALL.xml, it should look like something like this:
<rules>
<rule>
<key>SCITOOLS_0</key>\>
<name><![CDATA[SciTools Check: SciTools' Recommended Checks\"Commented Out" Code]]></name>
<description><![CDATA[SciTools Check (0): Description:  Sections of code should not be "commented out".]]></description>
<descriptionFormat>HTML</descriptionFormat>
<severity>BLOCKER</severity>
<cardinality>SINGLE</cardinality>
<status>BETA</status>
<type>BUG</type>
</rule>
<rule>
<key>SCITOOLS_1</key>
<name><![CDATA[SciTools Check: SciTools' Recommended Checks\Definitions in Header Files]]></name>
<description><![CDATA[SciTools Check (1): Description:  Objects and Functions should not be defined in Header Files.]]></description>
<descriptionFormat>HTML</descriptionFormat>
<severity>BLOCKER</severity>
<cardinality>SINGLE</cardinality>
<status>BETA</status>
<type>BUG</type>
</rule>
-
Hint: Limitation of Script
You will notice that the ALL.xml declares all the Checks with a severity of BLOCKER, a status of BETA and a type of BUG. You may modify the script "create_sonarqube-cxx-external-rules.pl" to change how they are declared and/or edit the output XML manually. The possible values are respectively "INFO, MINOR, MAJOR (default), CRITICAL, BLOCKER" for severity, "BETA, READY (default), DEPRECATED." For status and "CODE_SMELL (default), BUG and VULNERABILITY" for type.
ALL.xml.key should look like something like this:
SCITOOLS_0%5CSciTools' Recommended Checks\"Commented Out" Code
SCITOOLS_1%5CSciTools' Recommended Checks\Definitions in Header Files
SCITOOLS_2%5CSciTools' Recommended Checks\Files too long
SCITOOLS_3%5CSciTools' Recommended Checks\Floating-point expressions shall not be directly or indirectly tested for equality or inequality.
SCITOOLS_4%5CSciTools' Recommended Checks\Functions Too Long
SCITOOLS_5%5CSciTools' Recommended Checks\Functions shall not be declared implicitly
SCITOOLS_6%5CSciTools' Recommended Checks\Goto Statements
SCITOOLS_7%5CSciTools' Recommended Checks\Macros shall not be #define'd or #undef'd within a block
SCITOOLS_8%5CSciTools' Recommended Checks\Magic Numbers
SCITOOLS_9%5CSciTools' Recommended Checks\Nested Comments
SCITOOLS_10%5CSciTools' Recommended Checks\Overly Complex Functions
SCITOOLS_11%5CSciTools' Recommended Checks\Trigraphs shall not be used

5. Installation of Rules in SonarQube

The next step is to import that xml file into the SonarQube installation. To do so, go to
http://<SONARQUBE>/ settings?category=c%2B%2B+%28community%29&subcategory=%282%29+code+analysis
(alternatively, go to http://<SONARQUBE>/, then click Administration (you must be log as an admin) (Top), then C++ Community (Left), The Code Analysis (Kind of on Top))
Your browser should look like this:
Sonar Qube Browser
Scroll down until you see "External Rules".
Copy/Paste the content of ALL.xml in the "External Rules" textbox (at the bottom of the page)
Past external rules
Do not forget to Click the "Save Code Analysis Settings" (Bottom Left)
Restart your SonarQube
After you restart, go to the "Rules" tab (admin mode), and type "SciTools" in the searchbox (Top Left), you should now see the SciTools Checks as Rules (around 640 of them).
You are done!!!! SonarQube is ready to receive your Analysis!!!!!

6. Running an SciTools CodeCheck from the Command line.

Before we go back to SonarQube integration, let me explain how you run "CodeCheck" by SciTools from the command line.
6.1. Export the checks you want to use from CodeCheck in a .ini file.
Let's say you want to only check the MISRA C 2012 standard, select those checks in codecheck
You now need to do the opposite that we did earlier, you need to save this configuration to a .ini file, using the "Export Configuration to file" (DO NOT USE "Export Selected Checks to text file)
Select and export only MISRA 2012
Name the file MISRA-C-2012.ini.
If you are not sure whether you clicked the right button, MISRA-C-2012.ini should look like:
[CodeCheckConfigNames]
misra%20c%202012%20complete\CheckState\Published%20Standards%5CMISRA%20C%202012%5CDirective%204.3%20Assembly%20language%20shall%20be%20encapsulated%20and%20isolated=2
misra%20c%202012%20complete\CheckState\Published%20Standards%5CMISRA%20C%202012%5CDirective%204.4%20Sections%20of%20code%20should%20not%20be%20%22commented%20out%22=2
misra%20c%202012%20complete\CheckState\Published%20Standards%5CMISRA%20C%202012%5CDirective%204.5%20Identifiers%20in%20the%20same%20name%20space%20with%20overlapping%20visibility%20should%20be%20typographically%20unambiguous=2
misra%20c%202012%20complete\CheckState\Published%20Standards%5CMISRA%20C%202012%5CDirective%204.10%20Precautions%20shall%20be%20taken%20in%20order%20to%20prevent%20the%20contents%20of%20a%20header%20file%20being%20included%20more%20than%20once=2
misra%20c%202012%20complete\CheckState\Published%20Standards%5CMISRA%20C%202012%5C2.1%20A%20project%20shall%20not%20contain%20unreachable%20code=2
misra%20c%202012%20complete\CheckState\Published%20Standards%5CMISRA%20C%202012%5C2.3%20A%20project%20shall%20not%20contain%20unused%20type%20declarations=2
-
6.2. Set up an Environment variable for your ROOT
Remember you set up a ROOT in your udb file. I defined a LIBFREENECT2_ROOT for the root of my project, pointing to C:\Users\steph_000\Desktop\DEMOS\SonarQube\demo_project\libfreenect2-master.
I must declare an environment variable named UND_NAMED_ROOT_LIBFREENECT2_ROOT (it is UND_NAMED_ROOT_ plus whatever the name of your root was).
So for me, it looks like this:
set UND_NAMED_ROOT_LIBFREENECT2_ROOT=C:\Users\steph_000\Desktop\DEMOS\SonarQube\demo_project\libfreenect2-master
6.3. Running CodeCheck
To run CodeCheck from the command line:
und codecheck MISRA-C-2012.ini MISRA-C-2012-Results -db libfreenect2-master.udb
where:
  • MISRA-C-2012.ini is the file you exported from SciTools.
  • MISRA-C-2012-Results is the directory in which the results will be created
  • libfreenect2-master.udb is your db file.
It may be a good idea to run the analysis before you run codecheck, but this is not necessary. If desired, use the following two commands (again, this is not necessary):
und analyze -db libfreenect2-master.udb
und codecheck MISRA-C-2012.ini MISRA-C-2012-Results -db libfreenect2-master.udb
After a successful analyze, you should see the following files in the MISRA-C-2012-Results directory:
  • CodeCheckResultByFile.txt
  • CodeCheckResultByTable.csv
  • CodeCheckResultByViolation.txt

7. Transforming the Results of CodeCheck into a SonarQube readable XML.

Once again, you will need a script provided by Emenda. The script name is "create_sonarqube-cxx-report-from-scitools-analyis.pl". It will transform the output of SciTools to something SonarQube can read:
Run the following command:
uperl create_sonarqube-cxx-report-from-scitools-analyis.pl .\MISRA-C-2012-Results\CodeCheckResultByTable.csv CodeCheckResultByTable.xml ALL.xml.key LIBFREENECT2_ROOT
where:
  • .\MISRA-C-2012-Results\CodeCheckResultByTable.csv is the csv file created by the SciTools analysis.
  • CodeCheckResultByTable.xml is a file that will be created and loaded by sonar-scanner
  • ALL.xml.key was created earlier when you first exported all checks from SciTools.
  • LIBFREENECT2_ROOT is the root name you defined earlier.

8. Setting up Sonar-Scanner

We now need to set up Sonar to scan our project. To do so, add a file named "sonar-project.properties" in your project root (the same that you defined as LIBFREENECT2_ROOT). This file should contain a line with sonar.cxx.other.reportPath pointing to the file that is created in the previous paragraph, i.e. CodeCheckResultByTable.xml.
Here is my project.properties file (what's important is the last line!):
# must be unique in a given SonarQube instance
sonar.projectKey=lib:free:kinect
# this is the name and version displayed in the SonarQube UI. Was mandatory prior to SonarQube 6.1.
sonar.projectName=Lib Free Kinect
sonar.projectVersion=2.0
# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows.
# Since SonarQube 4.2, this property is optional if sonar.modules is set.
# If not set, SonarQube starts looking for source code from the directory containing 
# the sonar-project.properties file.
sonar.sources=.
# Encoding of the source code. Default is default system encoding
#sonar.sourceEncoding=UTF-8
#Added for Project:
sonar.cxx.includeDirectories=./include,./include/internal
sonar.cxx.other.reportPath=./CodeCheckResultByTable.xml 

Well, this is it, you have everything in place. Now, if you want to run an analysis (thru Jenkins or others), here is the list of what you need to do:

a) Set you root:
set UND_NAMED_ROOT_LIBFREENECT2_ROOT=C:\Users\steph_000\Desktop\DEMOS\SonarQube\demo_project\libfreenect2-master
b) Analyze the project with SciTools
und analyze -db libfreenect2-master.udb
c) Run CodeCheck using your .ini file.
und codecheck MISRA-C-2012.ini MISRA-C-2012-Results -db libfreenect2-master.udb
d) Convert the results from scitools to SonarQube
uperl create_sonarqube-cxx-report-from-scitools-analyis.pl .\MISRA-C-2012-Results\CodeCheckResultByTable.csv CodeCheckResultByTable.xml ALL.xml.key LIBFREENECT2_ROOT
e) Run a sonar scan !!!
sonar-scanner
When done, you will be able to see the SciTools Issues in SonarQube:
You done!!!!!
Please send comments and questions to Stephane.Raynaud@Emenda.com

6. About the Author: Thanks for Reading.

Programmer joke
Stephane Raynaud, Principal at Emenda USA. Stephane is an engineer that decided to pursue a career in software development and consulting. His first job was developing requirements for a drone program and prior to Emenda he had been running a multi million dollar consulting organization around C and C++. He is now applying his expertise and Emenda Best Practices to help software developers across North America build better code. ( Stephane.Raynaud@Emenda.com )