CheckRule¶
This module create targets that runs and generate reports about unit-tests.
Prerequisites¶
- enable_testing()
- This module requires that
enable_testing()
is called at top level CMakeLists.txt. - xsltproc
- XSL Template rendering tool. Available from ubuntu packages or from source at http://xmlsoft.org/
Functions¶
add_check(<module>
[ PATTERNS <pattern> [<pattern> ...]]
[ INCLUDES <dir> [<dir> ...]]
[ LINKS <lib> [<lib> ...]]
[ ENV <key>=<value> [<key=value> ...]]
[ ARGS <arg> [<arg> ...]]
[ DIRECTORY <dir> ]
[ PREFIX <str> ]
[ JOBS <int> ]
[ CMAKEVARS_NAME <name> ]
[ NO_DEFAULT_ENV ]
[ NO_DEFAULT_ARGS ]
[ NO_DEFAULT_INCLUDES ]
[ NO_DEFAULT_LINKS ]
)
This function automatically detects tests source files, creates binary targets and generate test report.
Parameters¶
- module
- Name of the module. It determines the name of the generated cmake targets and the directory where targets generate the report.
- PATTERNS
List of file extensions to match while searching for tests. See details about how tests are automatically detected by this module.
Default value is given by
CheckRule_DEFAULT_PATTERNS
Warning
Items given in
PATTERNS
list are not wildcards but only file extensions (ie: no asterix)- INCLUDES
List of include directories to add when compiling test sources. Each item will be added through cmake
target_include_directories
directive.Warning
When using cmake version prior to 2.8.12, test include directories are added through cmake
include_directories
. Therefore, they will also be added to your CMakeLists.txt targets.Default value is given by
CheckRule_DEFAULT_INCLUDES
unlessNO_DEFAULT_INCLUDES
option is given.- LINKS
List of libraries to add when linking test binaries. Each item will be added through cmake
target_link_directories
directive.Default value is given by
CheckRule_DEFAULT_LINKS
unlessNO_DEFAULT_LINKS
option is given.- ENV
List of environment variable to defined before running each test.
Default value is given by
CheckRule_DEFAULT_ENV
unlessNO_DEFAULT_ENV
option is given.- ARGS
List of command-line options to pass when running test binaries.
Default value is given by
CheckRule_DEFAULT_ARGS
unlessNO_DEFAULT_ARGS
option is given.Tip
This option is a convenient way to give your tests some informations about source and build directory tree.
Default value is given
CheckRule_DEFAULT_ARGS
- DBG_ARGS
List of command-line options to pass when running test through debugger. It Usually sets arguments to command line to prevent your test framework to protect run with forks, allowing to get a usable frame-stack to investigate crashes.
Default value is given by
CheckRule_DEFAULT_DBG_ARGS
unlessNO_DEFAULT_ARGS
option is given.Default value is given
CheckRule_DEFAULT_DBG_ARGS
- DIRECTORY
Directory to search tests source files. See details about how tests are automatically detected by this module.
Default value is given
CheckRule_DEFAULT_DIRECTORY
- PREFIX
Filename prefix of test source files. See details about how tests are automatically detected by this module.
Default value is given
CheckRule_DEFAULT_PREFIX
- JOBS
Number of simultaneous test to run when target is called.
Default value is given
CheckRule_DEFAULT_JOBS
- CMAKEVARS_NAME
Path to header file generated by check rule. See details about how getting information about source/build tree in your test code.
Default value is given
CheckRule_DEFAULT_CMAKEVARS_NAME
- NO_DEFAULT_ENV
- If option is given, don’t use
CheckRule_DEFAULT_ENV
- NO_DEFAULT_ARGS
- If option is given, don’t use
CheckRule_DEFAULT_ARGS
- NO_DEFAULT_INCLUDES
- If option is given, don’t use
CheckRule_DEFAULT_INCLUDES
- NO_DEFAULT_LINKS
- If option is given, don’t use
CheckRule_DEFAULT_LINKS
Global variables¶
-
CheckRule_DEFAULT_PATTERNS
¶ -
".c;.cc;.cpp"
-
CheckRule_DEFAULT_INCLUDES
¶ -
""
-
CheckRule_DEFAULT_LINKS
¶ -
""
-
CheckRule_DEFAULT_ENV
¶ -
""
-
CheckRule_DEFAULT_DIRECTORY
¶ -
"${CMAKE_CURRENT_SOURCE_DIR}/unit"
-
CheckRule_DEFAULT_PREFIX
¶ -
"Test"
-
CheckRule_DEFAULT_JOBS
¶ -
"1"
-
CheckRule_DEFAULT_ARGS
¶ -
""
-
CheckRule_DEFAULT_DBG_ARGS
¶ -
""
-
CheckRule_DEFAULT_CMAKEVARS_NAME
¶ -
"${CMAKE_CURRENT_BINARY_DIR}/cmakevars.h"
-
CheckRule_DEFAULT_TIMEOUT
¶ -
"120"
Getting path informations in tests¶
Tests often need to read sample files located in either source or build directory. Because source and build trees are not relative to each others, only CMake knows where both root directories are located.
XTDMake’s CheckRule provides two ways to forward this informations to your tests :
Using
ARGS
and/orCheckRule_DEFAULT_ARGS
to add command line parameters built with CMake variables such as :
CMAKE_SOURCE_DIR
: top source directoryCMAKE_BINARY_DIR
: top build directoryCMAKE_CURRENT_SOURCE_DIR
: current module’s source directoryCMAKE_CURRENT_BINARY_DIR
: current module’s build directoryOne possible value for
CheckRule_DEFAULT_ARGS
could be:--topsrc-dir=\${CMAKE_PROJECT_SOURCE_DIR} \ --topbuild-dir=\${CMAKE_PROJECT_BINARY_DIR} \ --src-dir=\${CMAKE_SOURCE_DIR} \ --build-dir=\${CMAKE_BINARY_DIR}Using generated header file. CheckRule automatically creates for each module an header file named
CheckRule_DEFAULT_CMAKEVARS
orCMAKEVARS
arguments. This file is generated from the given template :#define TOP_SRCDIR "@CMAKE_SOURCE_DIR@" #define SRCDIR "@CMAKE_CURRENT_SOURCE_DIR@" #define TOP_BUILDDIR "@PROJECT_BINARY_DIR@" #define BUILDDIR "@CMAKE_CURRENT_BINARY_DIR@" #define PROJECT_SOURCE_DIR "@PROJECT_SOURCE_DIR@" #define PROJECT_BINARY_DIR "@PROJECT_BINARY_DIR@"Your test code can simply include the generated header and use defined variables to build path to your assets files located in source or build tree.
Finding the test sources¶
This module scans given DIRECTORY
for source files prefixed by PREFIX
and matches
one of file extensions given by PATTERNS
. Each matched file is considered as a standalone
executable test.
Deducing the target name¶
This function deduces the name of the test from its source file by stripping DIRECTORY
,
PREFIX
and match extension. Example :
file: ./unit/TestApplication.cc DIRECTORY: ./unit PATTERNS: .cc;.cpp.c Deduced name: Application
Generated targets¶
check
- generate doc reports for all modules
check-clean
- removes doc reports for all modules
<module>-check
- generate unittests report for module <module>
<module>-check-build
- build all test binaries for module <module>
<module>-check-run
- run tests for module <module> that are not up-to-date
<module>-check-run-verbose
- run tests for module <module> that are not up-to-date with ctest verbose output
<module>-check-run-forced
- run all tests for module <module>
<module>-check-clean
- clean test targets for module <module>
For each test <name>, the function also produces :
t<name>
- build individual test binary target <name>
<module>-check-ut-<name>
- run individual test <name>
<module>-check-ut-<name>-dbg
- run individual test <name> wrapped in debugger
<module>-check-ut-<name>-cmd
- prints individual test command <name>
Adding test manually¶
To integrate manually defined tests with CheckRule module, you must use the following function.
Warning
This function must be called before add_check
add_check_test(module name
COMMAND <command> [ <arg> ... ]
[ ENVIRONMENT <var>=<value> [ <var>=<value> ... ]
)
- module
- name of targeted module
- name
- name of the test target
- COMMAND
- command line to run for this test
- ENVIRONMENT
- environment variable to define before running the test
About debugger¶
By default, CheckRule debugger target wraps test execution in GNU gdb
. If USE_CLANG
variable is defined, debugger is switched to lldb
.
Target Dependencies¶
Warning
The dependency of cmake build system to the modification time of
DIRECTORY
doesn’t work with cmake versions prior to 3.0.
This mean you must re-run cmake after adding new sources files in
order to properly update the rule files dependencies.
Generated reports¶
HTML : reports/check/<module>/index.html
Bellow an example of generated html report :
XML : reports/check/<module>/index.xml
<?xml version="1.0" encoding="UTF-8"?>
<Site BuildName="(empty)"
BuildStamp="20161231-1237-Experimental"
Name="(empty)"
Generator="ctest-3.5.1"
CompilerName=""
CompilerVersion=""
OSName="Linux"
Hostname="PSYCO-INTEL"
OSRelease="4.4.0-57-generic"
OSVersion="#78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016"
OSPlatform="x86_64"
Is64Bits="1"
VendorString="GenuineIntel"
VendorID="Intel Corporation"
FamilyID="6"
ModelID="79"
ProcessorCacheSize="20480"
NumberOfLogicalCPU="16"
NumberOfPhysicalCPU="1"
TotalVirtualMemory="93"
TotalPhysicalMemory="64340"
LogicalProcessorsPerPhysical="16"
ProcessorClockFrequency="1898.75"
>
<Testing>
<StartDateTime>Dec 31 13:37 CET</StartDateTime>
<StartTestTime>1483187874</StartTestTime>
<TestList>
<Test>./tApplication</Test>
</TestList>
<Test Status="passed">
<Name>tConfigParser</Name>
<Path>.</Path>
<FullName>./tConfigParser</FullName>
<FullCommandLine>/home/psyco/dev/xtdcpp/.release/core/tConfigParser "--srcdir=/home/psyco/dev/xtdcpp/core" "--top-srcdir=/home/psyco/dev/xtdcpp" "--top-builddir=/home/psyco/dev/xtdcpp/.release" "--testdir=/home/psyco/dev/xtdcpp/core/unit" "--outputter=compiler" "-p" "-e" "7"</FullCommandLine>
<Results>
<NamedMeasurement type="numeric/double" name="Execution Time">
<Value>0.0134299</Value>
</NamedMeasurement>
<NamedMeasurement type="text/string" name="Completion Status">
<Value>Completed</Value>
</NamedMeasurement>
<NamedMeasurement type="text/string" name="Command Line">
<Value>/home/psyco/dev/xtdcpp/.release/core/tConfigParser "--srcdir=/home/psyco/dev/xtdcpp/core" "--top-srcdir=/home/psyco/dev/xtdcpp" "--top-builddir=/home/psyco/dev/xtdcpp/.release" "--testdir=/home/psyco/dev/xtdcpp/core/unit" "--outputter=compiler" "-p" "-e" "7"</Value>
</NamedMeasurement>
<Measurement>
<Value>
TestConfParser::Constructor : start
TestConfParser::Constructor : end Ok
TestConfParser::parse : start
TestConfParser::parse : end Ok
TestConfParser::get : start
TestConfParser::get : end Ok
TestConfParser::search : start
TestConfParser::search : end Ok
TestConfParser::setParams : start
TestConfParser::setParams : end Ok
TestConfParser::parseFile : start
TestConfParser::parseFile : end Ok
OK (6)
</Value>
</Measurement>
</Results>
</Test>
<EndDateTime>Dec 31 13:37 CET</EndDateTime>
<EndTestTime>1483187875</EndTestTime>
<ElapsedMinutes>0</ElapsedMinutes>
</Testing>
</Site>
JSON : reports/check/<module>/status.json
{
"status": "success",
"graphs": [
{
"data": {
"labels": [],
"datasets": [
{
"borderColor": "rgba(51, 204, 51, 0.5)",
"pointBorderColor": "rgba(31, 122, 31, 1)",
"yAxisID": "absolute",
"label": "success tests",
"backgroundColor": "rgba(51, 204, 51, 0)",
"pointBackgroundColor": "rgba(31, 122, 31, 1)",
"data": "%(success)d"
},
{
"borderColor": "rgba(179, 0, 0, 0.5)",
"pointBorderColor": "rgba(102, 0, 0, 1)",
"yAxisID": "absolute",
"label": "failure tests",
"backgroundColor": "rgba(179, 0, 0, 0)",
"pointBackgroundColor": "rgba(102, 0, 0, 1)",
"data": "%(failures)d"
}
]
},
"type": "line",
"options": {
"scales": {
"xAxes": [
{
"ticks": {
"fontSize": 12,
"minRotation": 80
}
}
],
"yAxes": [
{
"position": "left",
"ticks": {
"fontSize": 24,
"beginAtZero": true
},
"type": "linear",
"id": "absolute",
"display": true
}
]
},
"title": {
"text": "%(module)s : unittests",
"display": true
}
}
}
],
"data": {
"failures": 0,
"success": 14
},
"label": "14 / 14"
}