CheckRule

This module create targets that runs and generate reports about unit-tests.

Prerequisites

enable_testing()

This module requires that enable_testing() is called at top level CMakeLists.txt.

xsltproc

XSL Template rendering tool. Available from ubuntu packages or from source at http://xmlsoft.org/

Functions

add_check(<module>
  [ PATTERNS  <pattern>     [<pattern>   ...]]
  [ INCLUDES  <dir>         [<dir>       ...]]
  [ LINKS     <lib>         [<lib>       ...]]
  [ ENV       <key>=<value> [<key=value> ...]]
  [ ARGS      <arg>         [<arg>       ...]]
  [ DIRECTORY      <dir>  ]
  [ PREFIX         <str>  ]
  [ JOBS           <int>  ]
  [ CMAKEVARS_NAME <name> ]
  [ NO_DEFAULT_ENV        ]
  [ NO_DEFAULT_ARGS       ]
  [ NO_DEFAULT_INCLUDES   ]
  [ NO_DEFAULT_LINKS      ]
)

This function automatically detects tests source files, creates binary targets and generate test report.

Parameters

module

Name of the module. It determines the name of the generated cmake targets and the directory where targets generate the report.

PATTERNS

List of file extensions to match while searching for tests. See details about how tests are automatically detected by this module.

Default value is given by CheckRule_DEFAULT_PATTERNS

Warning

Items given in PATTERNS list are not wildcards but only file extensions (ie: no asterix)

INCLUDES

List of include directories to add when compiling test sources. Each item will be added through cmake target_include_directories directive.

Warning

When using cmake version prior to 2.8.12, test include directories are added through cmake include_directories. Therefore, they will also be added to your CMakeLists.txt targets.

Default value is given by CheckRule_DEFAULT_INCLUDES unless NO_DEFAULT_INCLUDES option is given.

LINKS

List of libraries to add when linking test binaries. Each item will be added through cmake target_link_directories directive.

Default value is given by CheckRule_DEFAULT_LINKS unless NO_DEFAULT_LINKS option is given.

ENV

List of environment variable to defined before running each test.

Default value is given by CheckRule_DEFAULT_ENV unless NO_DEFAULT_ENV option is given.

ARGS

List of command-line options to pass when running test binaries.

Default value is given by CheckRule_DEFAULT_ARGS unless NO_DEFAULT_ARGS option is given.

Tip

This option is a convenient way to give your tests some informations about source and build directory tree.

Default value is given CheckRule_DEFAULT_ARGS

DBG_ARGS

List of command-line options to pass when running test through debugger. It Usually sets arguments to command line to prevent your test framework to protect run with forks, allowing to get a usable frame-stack to investigate crashes.

Default value is given by CheckRule_DEFAULT_DBG_ARGS unless NO_DEFAULT_ARGS option is given.

Default value is given CheckRule_DEFAULT_DBG_ARGS

DIRECTORY

Directory to search tests source files. See details about how tests are automatically detected by this module.

Default value is given CheckRule_DEFAULT_DIRECTORY

PREFIX

Filename prefix of test source files. See details about how tests are automatically detected by this module.

Default value is given CheckRule_DEFAULT_PREFIX

JOBS

Number of simultaneous test to run when target is called.

Default value is given CheckRule_DEFAULT_JOBS

CMAKEVARS_NAME

Path to header file generated by check rule. See details about how getting information about source/build tree in your test code.

Default value is given CheckRule_DEFAULT_CMAKEVARS_NAME

NO_DEFAULT_ENV

If option is given, don’t use CheckRule_DEFAULT_ENV

NO_DEFAULT_ARGS

If option is given, don’t use CheckRule_DEFAULT_ARGS

NO_DEFAULT_INCLUDES

If option is given, don’t use CheckRule_DEFAULT_INCLUDES

NO_DEFAULT_LINKS

If option is given, don’t use CheckRule_DEFAULT_LINKS

Global variables

CheckRule_DEFAULT_PATTERNS

".c;.cc;.cpp"

CheckRule_DEFAULT_INCLUDES

""

CheckRule_DEFAULT_LINKS

""

CheckRule_DEFAULT_ENV

""

CheckRule_DEFAULT_DIRECTORY

"${CMAKE_CURRENT_SOURCE_DIR}/unit"

CheckRule_DEFAULT_PREFIX

"Test"

CheckRule_DEFAULT_JOBS

"1"

CheckRule_DEFAULT_ARGS

""

CheckRule_DEFAULT_DBG_ARGS

""

CheckRule_DEFAULT_CMAKEVARS_NAME

"${CMAKE_CURRENT_BINARY_DIR}/cmakevars.h"

CheckRule_DEFAULT_TIMEOUT

"120"

Getting path informations in tests

Tests often need to read sample files located in either source or build directory. Because source and build trees are not relative to each others, only CMake knows where both root directories are located.

XTDMake’s CheckRule provides two ways to forward this informations to your tests :

1. Using ARGS and/or CheckRule_DEFAULT_ARGS to add command line parameters built with CMake variables such as :

   • CMAKE_SOURCE_DIR : top source directory
   • CMAKE_BINARY_DIR : top build directory
   • CMAKE_CURRENT_SOURCE_DIR : current module’s source directory
   • CMAKE_CURRENT_BINARY_DIR : current module’s build directory

   One possible value for CheckRule_DEFAULT_ARGS could be:

   --topsrc-dir=\${CMAKE_PROJECT_SOURCE_DIR} \
   --topbuild-dir=\${CMAKE_PROJECT_BINARY_DIR} \
   --src-dir=\${CMAKE_SOURCE_DIR} \
   --build-dir=\${CMAKE_BINARY_DIR}
   

2. Using generated header file. CheckRule automatically creates for each module an header file named CheckRule_DEFAULT_CMAKEVARS or CMAKEVARS arguments. This file is generated from the given template :

   #define   TOP_SRCDIR       "@CMAKE_SOURCE_DIR@"
   #define       SRCDIR       "@CMAKE_CURRENT_SOURCE_DIR@"
   #define TOP_BUILDDIR       "@PROJECT_BINARY_DIR@"
   #define     BUILDDIR       "@CMAKE_CURRENT_BINARY_DIR@"
   #define PROJECT_SOURCE_DIR "@PROJECT_SOURCE_DIR@"
   #define PROJECT_BINARY_DIR "@PROJECT_BINARY_DIR@"
   

   Your test code can simply include the generated header and use defined variables to build path to your assets files located in source or build tree.

Finding the test sources

This module scans given DIRECTORY for source files prefixed by PREFIX and matches one of file extensions given by PATTERNS. Each matched file is considered as a standalone executable test.

Deducing the target name

This function deduces the name of the test from its source file by stripping DIRECTORY, PREFIX and match extension. Example :

file:./unit/TestApplication.cc DIRECTORY:./unit PATTERNS:.cc;.cpp.c Deduced name:Application

Generated targets

check

generate doc reports for all modules

check-clean

removes doc reports for all modules

<module>-check

generate unittests report for module <module>

<module>-check-build

build all test binaries for module <module>

<module>-check-run

run tests for module <module> that are not up-to-date

<module>-check-run-verbose

run tests for module <module> that are not up-to-date with ctest verbose output

<module>-check-run-forced

run all tests for module <module>

<module>-check-clean

clean test targets for module <module>

For each test <name>, the function also produces :

t<name>

build individual test binary target <name>

<module>-check-ut-<name>

run individual test <name>

<module>-check-ut-<name>-dbg

run individual test <name> wrapped in debugger

<module>-check-ut-<name>-cmd

prints individual test command <name>

Adding test manually

To integrate manually defined tests with CheckRule module, you must use the following function.

Warning

This function must be called before add_check

add_check_test(module name
  COMMAND <command> [ <arg> ... ]
  [ ENVIRONMENT <var>=<value> [ <var>=<value> ... ]
)

module

name of targeted module

name

name of the test target

COMMAND

command line to run for this test

ENVIRONMENT

environment variable to define before running the test

About debugger

By default, CheckRule debugger target wraps test execution in GNU gdb. If USE_CLANG variable is defined, debugger is switched to lldb.

Target Dependencies

Warning

The dependency of cmake build system to the modification time of DIRECTORY doesn’t work with cmake versions prior to 3.0. This mean you must re-run cmake after adding new sources files in order to properly update the rule files dependencies.

Generated reports

HTML : reports/check/<module>/index.html

Bellow an example of generated html report :

XML : reports/check/<module>/index.xml

<?xml version="1.0" encoding="UTF-8"?>
<Site BuildName="(empty)"
      BuildStamp="20161231-1237-Experimental"
      Name="(empty)"
      Generator="ctest-3.5.1"
      CompilerName=""
      CompilerVersion=""
      OSName="Linux"
      Hostname="PSYCO-INTEL"
      OSRelease="4.4.0-57-generic"
      OSVersion="#78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016"
      OSPlatform="x86_64"
      Is64Bits="1"
      VendorString="GenuineIntel"
      VendorID="Intel Corporation"
      FamilyID="6"
      ModelID="79"
      ProcessorCacheSize="20480"
      NumberOfLogicalCPU="16"
      NumberOfPhysicalCPU="1"
      TotalVirtualMemory="93"
      TotalPhysicalMemory="64340"
      LogicalProcessorsPerPhysical="16"
      ProcessorClockFrequency="1898.75"
      >
      <Testing>
              <StartDateTime>Dec 31 13:37 CET</StartDateTime>
              <StartTestTime>1483187874</StartTestTime>
              <TestList>
                      <Test>./tApplication</Test>
              </TestList>
              <Test Status="passed">
                      <Name>tConfigParser</Name>
                      <Path>.</Path>
                      <FullName>./tConfigParser</FullName>
                      <FullCommandLine>/home/psyco/dev/xtdcpp/.release/core/tConfigParser "--srcdir=/home/psyco/dev/xtdcpp/core" "--top-srcdir=/home/psyco/dev/xtdcpp" "--top-builddir=/home/psyco/dev/xtdcpp/.release" "--testdir=/home/psyco/dev/xtdcpp/core/unit" "--outputter=compiler" "-p" "-e" "7"</FullCommandLine>
                      <Results>
                              <NamedMeasurement type="numeric/double" name="Execution Time">
                                      <Value>0.0134299</Value>
                              </NamedMeasurement>
                              <NamedMeasurement type="text/string" name="Completion Status">
                                      <Value>Completed</Value>
                              </NamedMeasurement>
                              <NamedMeasurement type="text/string" name="Command Line">
                                      <Value>/home/psyco/dev/xtdcpp/.release/core/tConfigParser "--srcdir=/home/psyco/dev/xtdcpp/core" "--top-srcdir=/home/psyco/dev/xtdcpp" "--top-builddir=/home/psyco/dev/xtdcpp/.release" "--testdir=/home/psyco/dev/xtdcpp/core/unit" "--outputter=compiler" "-p" "-e" "7"</Value>
                              </NamedMeasurement>
                              <Measurement>
                                      <Value>
            TestConfParser::Constructor : start
            TestConfParser::Constructor : end Ok
            TestConfParser::parse : start
            TestConfParser::parse : end Ok
            TestConfParser::get : start
            TestConfParser::get : end Ok
            TestConfParser::search : start
            TestConfParser::search : end Ok
            TestConfParser::setParams : start
            TestConfParser::setParams : end Ok
            TestConfParser::parseFile : start
            TestConfParser::parseFile : end Ok
            OK (6)
          </Value>
                              </Measurement>
                      </Results>
              </Test>
              <EndDateTime>Dec 31 13:37 CET</EndDateTime>
              <EndTestTime>1483187875</EndTestTime>
              <ElapsedMinutes>0</ElapsedMinutes>
      </Testing>
</Site>

JSON : reports/check/<module>/status.json

{
  "status": "success",
  "graphs": [
    {
      "data": {
        "labels": [],
        "datasets": [
          {
            "borderColor": "rgba(51, 204, 51, 0.5)",
            "pointBorderColor": "rgba(31, 122, 31, 1)",
            "yAxisID": "absolute",
            "label": "success tests",
            "backgroundColor": "rgba(51, 204, 51, 0)",
            "pointBackgroundColor": "rgba(31, 122, 31, 1)",
            "data": "%(success)d"
          },
          {
            "borderColor": "rgba(179, 0, 0, 0.5)",
            "pointBorderColor": "rgba(102, 0, 0, 1)",
            "yAxisID": "absolute",
            "label": "failure tests",
            "backgroundColor": "rgba(179, 0, 0, 0)",
            "pointBackgroundColor": "rgba(102, 0, 0, 1)",
            "data": "%(failures)d"
          }
        ]
      },
      "type": "line",
      "options": {
        "scales": {
          "xAxes": [
            {
              "ticks": {
                "fontSize": 12,
                "minRotation": 80
              }
            }
          ],
          "yAxes": [
            {
              "position": "left",
              "ticks": {
                "fontSize": 24,
                "beginAtZero": true
              },
              "type": "linear",
              "id": "absolute",
              "display": true
            }
          ]
        },
        "title": {
          "text": "%(module)s : unittests",
          "display": true
        }
      }
    }
  ],
  "data": {
    "failures": 0,
    "success": 14
  },
  "label": "14 / 14"
}