Code Coverage for Mendix apps

Introduction

When testing a Mendix app, you need to check and confirm the quality, functionality, and reliability of your app. You can test your Mendix app at different levels for these criterias, such as unit testing, component testing, process testing, and UI testing. Besides this distinction in test levels, there are two kinds of test approaches: white-box testing and black-box testing. White-box testing involves testing the internal structure and logic of the code, while black box testing checks whether your system is working according to the business requirement and end-users’ expectations without considering its programming or the internal functionalities.?

White box testing in Mendix means testing at the microflow and domain model level. Black box testing in Mendix is done through the UI and/or available API’s.

The white box and black box approaches also have two different coverage concepts. White box testing is linked to code coverage and black box testing to test coverage. Code coverage and test coverage are often mistaken or used interchangeably, but since they are linked to different test approaches they are not the same.

In this blog we explain the differences between code coverage and test coverage and how you can apply and measure code coverage in a Mendix application. In the last part of this blog we provide guidelines that can help you to determine how much code coverage you need for your Mendix app.

Difference between Code Coverage and Test Coverage

Code coverage and test coverage are two different but complementary metrics that can be used to evaluate the effectiveness and completeness of testing your Mendix app. Code coverage measures how many elements of the Mendix model are executed by the test cases, while test coverage measures how much of the software requirements or specifications are verified by the test cases. Both code coverage and test coverage are important for software quality assurance, as they can help identify where the system is tested and more importantly where it is NOT tested, and provide guidance for improvement and optimization.

Code Coverage

Code coverage is a measure of how many elements of the Mendix model are executed or covered by a set of test cases. Code coverage can be calculated at different levels of granularity, such as function coverage, statement coverage, exit coverage, branch coverage, condition coverage, etc. Code coverage can be expressed as a percentage of the total number of code elements (such as statements, exits, branches, functions, or conditions) that are covered by the test cases over the total number of code elements in the software product.

Code coverage can be used to assess the quality and completeness of white-box testing, as it indicates how well the internal structure and logic of the code are tested. Code coverage can also help identify the areas of the code that are not tested or have low coverage, and thus require more attention or improvement.?

While test coverage can be measured by platform independent tools, code coverage of Mendix apps can only be measured by specific tools that fit well with the Mendix platform.

Test Coverage

Test coverage is a measure of how much of the software requirements or specifications are verified or covered by a set of test cases. Test coverage can be calculated at different levels of abstraction, such as feature coverage, scenario coverage, use case coverage, or user story coverage. Test coverage can be expressed as a percentage of the total number of requirements or specifications that are covered by the test cases over the total number of requirements or specifications in the software product.

Test coverage can be used to assess the quality and completeness of black-box testing, as it indicates how well the external behavior and functionality of the software are tested. Test coverage can also help identify the gaps or discrepancies between the software requirements or specifications and the actual software product, and thus require more clarification or modification. Test coverage can be measured using various methods and tools, such as traceability matrix, test design, or test management.

Why is code coverage important?

The right level of code coverage in your automated tests helps you to:

• improve the quality and reliability of your Mendix app by ensuring that the code meets the expected behavior and functionality.
• reduce the risk of introducing errors or regressions when modifying or adding new features to the Mendix app.
• facilitate the maintenance and evolution of the Mendix app by making it easier to refactor, debug, and optimize the Mendix model.

Types of Code Coverage

Code coverage can be classified into different types. Some of the common types of code coverage that are useful in the context of Mendix apps are:

• Function coverage: the metric that measures how many functions in a program are executed during a test run.?
• Exit coverage: This measures the percentage of exit points or return statements in the source code that are reached by the test cases. An exit point is a location where a function or a program terminates its execution and returns control to the caller. Exit coverage is a type of statement coverage that focuses on the end of a function or a program, as it ensures that all the possible ways of exiting a function or a program are tested.
• Branch coverage: This measures the percentage of branches or decision points in the source code that are executed by the test cases. A branch is a point where the control flow can diverge based on a condition, such as an if-else statement or a switch-case statement. Branch coverage is more insightful than statement coverage, as it ensures that all the possible outcomes of a branch are tested.
• Statement coverage: This measures the percentage of executable statements in the source code that are executed by the test cases. It is the simplest and most basic type of code coverage, but it may not account for the different control paths or conditions that affect the execution of the statements. Therefore it is questionable how useful it really is, see also this blog about “What is Wrong with Statement Coverage”??

There are also different types of code coverage that are derived from or related to the above types, such as path coverage, condition coverage, loop coverage, parameter value coverage, etc. However, these types are less useful in the context of Mendix apps. The choice of the type of code coverage depends on the objectives, scope, and resources of your testing process, as well as the characteristics and requirements of the Mendix app. In general, higher levels of code coverage imply higher levels of code quality, but they also require more effort and cost to achieve. Therefore, it is important to balance the benefits and challenges of code coverage and use it as a relative and indicative measure rather than an absolute and definitive one.

Code coverage applied to Mendix

In Mendix we define a microflow as a “function” and a microflow activity as a “statement. The “exit” is the endpoint of a microflow (with or without return parameter).

A “branch” in a Mendix model is the most complex to define. There are explicit and implicit branches in Mendix models. Explicit branches are decision points in a microflow, but implicit branches in the logic of the Mendix model could be created by:?

• “If then else” or other logical statements in a microflow activity?
• event handlers on the domain model?
• (custom) Java actions in a microflow

To identify implicit branches in Mendix models? it is necessary to analyze the model elements that you want to test in depth. A Mendix model therefore becomes much more testable if the number of implicit branches is kept to the minimum. Testing becomes also easier if implicit branches are properly documented in the code.

We consider it not to be useful to measure statement coverage in a Mendix app at the application level. The microflow activities are provided by the Mendix platform and are tested already by Mendix. The only exception is for custom java activities. These, however, can be unit tested independently from the microflow in which they are used.

REMARK: Other types of functions in Mendix are nanoflows and object actions. However, nanoflows are not easy to execute separately and only exist in the context of a client. If a nanoflow wants to change something on the Mendix backend, it can only do this by calling a microflow or executing an object action.? Object actions are completely tested by the Mendix platform and do not need additional testing at the application level.?

Example of code coverage in Mendix on a single microflow

To make code coverage more concrete, we can apply the different types of code coverage to a specific microflow.

In the microflow above, which we also use as an example in our demo videos for showing how to build unit tests, we can define function, exit, branch and statement coverage as follows:

• Function coverages measures if the microflow is executed at all
• Exit coverage measures which of the four exits have been triggered by applying datavariation to the input of the test
• Branch coverage measures which of the twelve branches have been triggered by applying datavariation to the input of the test
• Statement coverage measures which microflow activities have been executed.?

In the microflow above no implicit branches are present. There are only simple true or false statements in the decisions and only the last decision (Fuel?) uses an enumeration.

The same microflow could also have been built with the implicit branches in the decision Expression (see image below)

At first sight, such a microflow would look simpler than the one above, but because of the implicit branching it is much harder to test in a unit test.

What level of code coverage makes sense for each test level?

Function coverage is the starting point of Mendix code coverage, since it can be measured at all test levels (unit, component and process). Exit and branch coverage are also useful metrics, specifically to get a better idea how well you have tested the logic within microflows. For unit tests (simple microflows) it? is desirable to achieve 100% exit coverage and if possible also 100% branch coverage by varying the input data to make sure that their behavior is controlled under any circumstances.?

For more complex microflows (with multiple branches and exits) that behave more like a component or process there are two ways of dealing with exit and branch code coverage

• The microflow is left as it is and is only tested with limited exit and branch coverage.?
• The microflow is split into submicroflows with a lower number of exits and branches per submicroflow. These submicroflows can then be tested as units with 100% exit and branch coverage.

In an earlier blog we already wrote about how to increase code coverage with less test cases?

In a future blog we will also address tactics on how and when to split a microflow into submicroflows. Not only to make them more testable but also to structure the microflows in? well-defined ways that makes them more maintainable and reusable.

How to measure code coverage in Mendix during a testrun?

Each test case in a test run can execute one or more microflows. Some of them are executed explicitly in a teststep. Other microflows are implicitly executed as a submicroflow.

Therefore, if you want to measure microflow coverage? in a Mendix app, you need to measure all executed microflows in a test run (and not only the ones that were part of your test case definition).

Branch coverage and exit coverage can (at this moment) only? be measured manually during the test run within Mendix StudioPro by using the built in debugging functionality for microflows. Adding breakpoints to the exit paths and branches makes this easy for explicit branches in a microflow.?

Measuring branch coverage for implicit branches in a microflow is a lot harder, because you cannot set a breakpoint in Mendix StudioPro at a specific point of your code within a microflow activity. If you use java functions in a microflow it is possible to debug these java functions, but this requires a specific java debugging setup for Mendix

How much code coverage do you need?

There is no definitive answer to how much code coverage is needed for a Mendix app, as it depends on various factors such as the complexity, size, type, and purpose of the app, the testing methodology and tools used, the available resources and time, and the quality standards and expectations of the stakeholders.

However, some general guidelines and best practices can be followed to determine an appropriate level of code coverage for a software project, such as:

• Setting a minimum goal of code coverage that is acceptable for the project, based on the project requirements and specifications, and ensuring that the executed tests meet or exceed that goal.
• Using multiple types and levels of testing, such as unit, component and process testing, to cover different aspects and scenarios of the software functionality and behavior.
• Using code coverage tools and metrics that are suitable for the Mendix platform, that provide meaningful and accurate feedback on the code coverage. For example, focus on function coverage on all test levels, and only focus on branch/exit coverage for unit tests.
• Using code coverage as a complement, not a substitute, for other quality assurance techniques, such as code reviews, static analysis, manual testing, and user feedback.
• Continuously monitoring and improving the code coverage as the software evolves, by adding, updating, and removing tests as needed, and by addressing any gaps or issues identified by the code coverage analysis.

Sometimes it is hard to set one coverage goal for an application. Multiple coverage goals can be set for example by:

• Analyzing the domain model and set goals on testing the interaction of the Mendix model with the most risky entities and attributes
• Focussing on the microflows of specific modules or functionalities where the risk is highest
• Focussing on risky microflow (e.g. microflows with complex behavior)
• Focussing on those parts of the model that are frequently changed or used

In such a case it is necessary that you can measure the coverage for each of these goals.

Conclusion

In this blog we have described what code coverage is, how it differs from test coverage and how you can apply it to Mendix apps. Code coverage is certainly a useful concept for Mendix apps as an addition to other quality measures. Although it seems logical to see coverage as the measure of quality, the reverse is actually even more true, because the insight into what has NOT been tested can be more important than what has been tested.?

In a future blog we will explore how you can use the results of code coverage in your organization to increase the quality of your tests and therefore of your Mendix application.

This blog is also published on the Menditect website https://menditect.com/code-coverage-for-mendix-apps/