Improving the Verification Process using the UVVM Specification Coverage VIP

In today’s technology-driven world, where electronic systems are becoming increasingly complex, effective verification is essential to ensure that designs function as expected. The Universal VHDL Verification Methodology (UVVM) is a well known framework for VHDL verification, and a central component of this framework is the Specification Coverage Verification IP (VIP). In this article, I will explain what the Specification Coverage VIP is and how it is used in UVVM.

Prioritising requirement verification

The Specification Coverage VIP in UVVM is a verification tool aimed at measuring requirement compliance and functional coverage, rather than focusing on code coverage. This means that the VIP concentrates on ensuring all functional requirements and specifications are verified and tested according to the design requirements. The benefit of utilising this VIP is that it provides a more focused verification approach, clearly identifying which requirements are met and which lack coverage.

Using the Specification Coverage VIP begins with defining a requirements list of all functional requirements and specifications that the design must comply with. This list is maintained as a simple CSV file and may include associated test cases.

Implementing the Specification Coverage VIP into the UVVM testbench

Implementing the VIP into the UVVM testbench is done by using the methods initialize_req_cov(), tick_off_req_cov(), and finalize_req_cov().

initialize_req_cov("T_UART_DEFAULTS", C_REQ_LIST_FILE, C_PARTIAL_COV_FILE);

-- run test to verify requirement(s)
sbi_check(SBI_VVCT, 1, C_ADDR_RX_DATA, x"00", "RX_DATA default");
await_completion(SBI_VVCT,1,  10 * C_CLK_PERIOD);
-- log requirement tick off after test
tick_off_req_cov("FPGA_SPEC_1.a");

sbi_check(SBI_VVCT, 1, C_ADDR_TX_READY, x"01", "TX_READY default");
await_completion(SBI_VVCT,1,  10 * C_CLK_PERIOD);
-- log requirement tick off after test
tick_off_req_cov("FPGA_SPEC_1.b");

sbi_check(SBI_VVCT, 1, C_ADDR_RX_DATA_VALID, x"00", "RX_DATA_VALID default");
await_completion(SBI_VVCT,1,  10 * C_CLK_PERIOD);
-- log requirement tick off after test
tick_off_req_cov("FPGA_SPEC_1.c");


-- end test case requirement coverage process
finalize_req_cov(VOID);        

Each test case will update the requirement status by marking off the requirements that have been verified and logging the results as shown above, ensuring a detailed and consistent tracking of each requirement throughout the complete verification process. Partial coverage files are generated every time test cases are run, showing which requirements have been tested and the result. These files are important as they are used for creating a complete coverage report.

After the test case runs, the Python script run_spec_cov.py is used to evaluate the test case results and generate a specification coverage report that provides an overall status of requirement compliance.

python3 ../script/run_spec_cov.py -h

run_spec_cov command line arguments (see QuickReference for more details):

-r                  Path/requirements.csv contains requirements
-p                  Path/testcase_result.csv partial coverage file from VHDL simulations
-m                  Optional: path/subrequirements.csv requirement map file
-s                  Path/spec_cov.csv specification coverage file
--strictness N      Optional: will set specification coverage to strictness (1-2) (0=default)
--config            Optional: configuration file with all arguments
--clean (DIR)       Will clean any/all partial coverage file(s) (default=current directory)
--help              This help screen        

The generated reports allow you to identify any gaps in requirement coverage and update the test cases as necessary to cover requirements that were missed before.

Configuration:
----------------------------------------------
requirement_list     : ../doc/req/req_list.csv
partial_cov          : ../doc/req/partial_cov_list.txt
requirement_map_list : ../doc/req/req_map.csv
spec_cov             : ../sim/spec_cov.csv
clean                : None
strictness           : 0
config               : None


SUMMARY:
----------------------------------------------
Number of compliant requirements     : 7
Number of non compliant requirements : 0
Number of non verified requirements  : 0
Number of not listed requirements    : 0
Number of passing testcases : 4
Number of failing testcases : 0
Number of not run testcases : 0


Compliant requirement(s) :
FPGA_SPEC_1.a, FPGA_SPEC_1.b, FPGA_SPEC_1.c, FPGA_SPEC_2, FPGA_SPEC_3, FPGA_SPEC_4, FPGA_SPEC_1, 

Passing testcase(s) :
T_UART_DEFAULTS, T_UART_TX, T_UART_RX, T_UART_SIMULTANEOUS,         

Importance in safety-critical applications

The Specification Coverage VIP provides a complete overview of which functional requirements have been tested and met, which is particularly important in safety-critical applications. In industries like subsea, medical technology, and space, where the reliability of electronic systems is extremely important, having a tool that ensures all specifications are thoroughly tested is invaluable. Having requirement traceability is not just a regulatory necessity but a critical component in delivering safe and effective products.

In designs that need to comply with the DO-254 safety standard, a well known safety standard for electronic hardware in the aviation industry that requires extensive verification and documentation, the Specification Coverage VIP offers considerable benefits. It provides direct traceability between specification requirements and test cases, which is essential for meeting DO-254’s requirements for traceability from requirements to tests. The detailed reports generated by the VIP can be used in documentation to demonstrate that all requirements have been tested and met.

Summary

Inventas not only develop UVVM but also actively use it along with the Specification Coverage VIP in several space, subsea and medical technology projects. These industries demand the highest levels of reliability and safety, and requirement traceability is an important component in delivering safe and effective products.