Getting Started with Code Coverage: Using CodeCov

“Is a 100% code coverage a metric for reliability and code quality?”

Well, I didn't intend to answer this question, but I will anyway. However, in this article, I will mainly walk you through how to utilize a tool called?CodeCov.

CodeCov?is a coverage solution platform that allows you to visualize your code coverage reports. CodeCov provides an in-depth coverage insights and calculates the hit-and-miss rate per file. It integrates easily with GitHub and GitLab. CodeCov also syncs all repositories (public and private). Of course, you have to manually setup CodeCov onto your repository the first time, which I will walk you through below.

But first:

Does a higher code coverage percentage means a well reliable and tested codebase?

In my own opinion having a?high?coverage percentage means that the entrypoint of the coverage report has reached the highest number possible of hit rate on lines of your code. By?entrypoint, I mean the set of instructions you used to run a coverage report. In most cases,?tests?are the entrypoints of most coverage reports. Tests are used to try to simulate an environment where it is plausible to execute your codebase.?Therefore, the higher the percentage of code coverage is a measurement of your entrypoints having hit a high number of code lines.?Does it measure reliability, security or compatibility? Not necessarily— it is a metric to understand how much coverage is reached by the tests of your codebase.?So, let’s setup codecov and put things in perspective.

Getting Started

Imagine you have a project that you are working on, where there are unit tests that you want to visualize. The visualization describes how much of the code lines do the unit tests actually hit. In this example, I will be using?Python?and two main libraries,?Coverage?and?CodeCov. To make this easier, I will also be using this?sample project?and I suggest reading this article below before.

The CodeCov visualization and insights can be automated into your development process. However, you will need to have continuous integration pipelines to automate generating your coverage reports and uploading them to CodeCov—and include a coverage badge too.

Requirements

1. An existing GitHub Repository
2. CodeCov Account

Using the?continuous_integration?repo mentioned in the Medium?article?above, I will write down the steps used to integrate CodeCov onto the repo.

Coverage Walkthrough

Add the two main python library needed using poetry, (you can use pip of course, but you really should check out?Poetry)

poetry add coverage codecov        

Then using?coverage?I will now run?simple_loop.py

coverage run simple_loop.py        

“Coverage.py?measures code coverage, typically during test execution. It uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed.”

This generates a series of files that will be used by CodeCov.

To create a CodeCov account, go to?CodeCov.io?and use login by?GitHub.

After you login, you will need to manually setup the repo on which you need CodeCov to use. I have already done this step using the?Not yet setup?button you see on the top right below in Fig 2.

So now, I instructed CodeCov to setup a Token and allow for an integration from my GitHub account specifically that repo. This information is available to you as soon as you click on the project link in CodeCov.io Repo section as you will see something like in Fig 3 below.

Going back to the project you can now upload your commit files using the?codecov?library and specifying the project by using the?codecov_token. Follow the red line in Fig 3 to find your?token.

codecov --token={codecode_token}        

CodeCov with GitHub Workflow

Well, you shouldn’t manually runcoverage?and?codecov?locally, remember this is supposed to run on test cases in your pipelines or workflows. Therefore, this should be integrated into you testing pipeline or testing stage within your continuous integration workflow. Using?GitHub?workflow as an example, I will make a custom workflow file with one job to run both coverage and codecov.

First, you need to store that token as a repository secret in your GitHub repository’ settings under secrets under actions.

Second, add a workflow job that runs your coverage and codecov commands.

# Github workflow runner instructions using cutomized action
name: Code Coverage [CodeCov]

on: [ push ]

jobs:
  codecov:
    runs-on: ubuntu-latest
    name: CodeCov
    steps:
      ##########   CHECK OUT REPO AND DOWNLOAD PACKAGES AND RUN COVERAGE & CODECOV #######
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install poetry and python packages
        run: |
          pip install poetry pyfiglet
          poetry config virtualenvs.create false
          poetry install
          pyfiglet Continuous Integration
      - name: Run Coverage
        run: coverage run simple_loop.py
      - name: Run CodeCov
        run: codecov --token=${{ secrets.codecov_token }}"s        

Finally, every push will now trigger a workflow named?Code Coverage [CodeCov]?responsible only for running the?simple_loop.py?python file using?coverage?and uploading the coverage data to CodeCov.io using the secret token in Fig 4. Check out the codecov public report for?continuous_integration?repo?here.

Coverage As a Metric

Let’s take the example of coverage on the?simple_loop.py?to understand what exactly happens.

""
A simple looping function with one python package requirement
"""
from tqdm import tqdm


def simple_loop_func(number):
    """
    Using tqdm progress bar this function prints out the progress of the for loop
    :param number: int digit to be the end of the for loop
    """
    for _ in tqdm(range(number)):
        pass

if __name__ == '__main__':
    print("This is a progress bar of the simple loop script")
    simple_loop_func(1000000)"        

Running?coverage run simple_loop.py?will execute the python script starting by the imports at line 4 and then main function, and finally calls the last line with?simple_loop_func(1000000)?function. Technically, the coverage hit rate of lines of code will be?100%.?To understand how does this?value?decrease, I have added another function with the purpose of being ignored.

""
A simple looping function with one python package requirement
"""
from tqdm import tqdm


def simple_loop_func(number):
    """
    Using tqdm progress bar this function prints out the progress of the for loop
    :param number: int digit to be the end of the for loop
    """
    for _ in tqdm(range(number)):
        pass


def __ignored_func():
    print("This is an ignored function and this message will not be printed")


if __name__ == '__main__':
    print("This is a progress bar of the simple loop script")
    simple_loop_func(1000000)"        

Well,?simple_loop.py?contains two functions. One function?simple_loop_func(number)prints out a progress bar and another intended for ignoring function?__ignored_func(). Not calling the function of?__ignored_func()?will not decrease the hit rate of the coverage, and the result on codecov will look like?that:

Now you can try answering the question: “?Is?a 100% code coverage a metric for reliability and code quality?”.

The sole intentions of calling direct scripts with coverage library was meant to replicate the behaviour of a testing unit or series of testing instructions. CodeCov offers a free package for up to 5 users, it also provides a badge with your coverage percentage and an SVG graph of your files.

Thanks!