I'll start with a confession: as a software engineer, I hated and avoided writing tests for years. I came across a lot of dusty projects: some without any tests, others that had tests but had never been run as part of CI/CD pipelines, and the last included really poor test coverage.
On the one hand, there are many reasons why we should write tests, but on the other, there are more excuses (or “justifications”) why we skip them.
While I aspired to be a professional, I was always jealous of those shiny open source repositories with 100% test coverage and dreamed about it in my day-to-day repositories. Four years ago, while wrestling with myself on this issue, I found differential covera great open source project with a simple mission: cover your own changes with testing. This is how the authors describe it:
Differential coverage is the percentage of new or modified lines that are covered by the tests. This provides a clear and achievable standard for code review: if you touch a line of code, that line should be covered. Code coverage is the responsibility of each developer!
Simply put, diff-cover assumes that you are using git and run a coverage tool. Using git it is easy to achieve his modified line numbers and compare them to the discovered line numbers from your favorite coverage tool. Almost all coverage tools can generate a unified and generic XML format, no matter what your coding language is (Python, JavaScript, etc.)
In summary, the process I have been doing so far as part of the CI was:
- Run all tests with a coverage tool, using the pytest and pytest-cov packages:
py.test -o junit_family=xunit2 --junitxml result.xml -xv --ff --cov-config=.coveragerc --cov=<my_package> --cov-report=xml --cov-report=term <tests_package>
(note that it will create coverage.xml and result.xml report files).
2. Run the differential cover tool, using the differential cover package:
diff-cover coverage.xml --compare-branch=origin/master
which will print something like the following output:
-------------
Diff Coverage
Diff: origin/master...HEAD, staged and unstaged changes
-------------
my_package/connections/snowflake_client.py (100%)
my_package/logic/celery_tasks/top_score_task.py (100%)
my_package/queries/build_algorithm_studio_dataframes.py (100%)
-------------
Total: 16 lines
Missing: 0 lines
Coverage: 100%
-------------
As you can see from the output above, I made changes to 3 different files and each of them is completely covered (I had to add some new tests and some of my changes were already covered by other existing tests).
Seeing that differential coverage report on every PR (Pull Request) has made everyone addicted to achieving that 100%. We want to show that we are responsible for our changes and can cover them, rather than being perceived as a loser and getting a low percentage. Also, as a side effect, we have experienced smaller incremental changes in PRs, which is another better practices. This is because everyone now thinks twice before adding redundant lines of code.
After using this methodology for a few years, we see a steady increase in the overall coverage percentages of our repositories. As a result, there has also been an increase in our production stability.
New GitHub Action
A few months ago, my talented colleague Asaf Gallea He decided to leverage this success into a simpler but more powerful solution. new GitHub action. This action applies the same idea as diff-cover and also generates a friendly report as a comment in your pull request, providing links to discovered lines in case you missed something. The new action also allows you to set a minimum coverage threshold (defaults to 80%); otherwise, the status check will fail and you won't be able to merge your changes:
In the image above we see the example of the GitHub action report. There is a minimum threshold of 95%, 20 lines were changed in this pull request, with 18 lines covered by tests and two lines, 505 to 506, are not covered. Since we only achieved 90% coverage for the modified files, the health check failed and it is impossible to merge them into the master branch.
Please note that this report does not say anything about the total coverage of the repository. It may be low (60%) and yet any new changes must exceed 95%, so eventually the total coverage will increase.
Configure the tests-coverage-report action in your repository
That's all! Now let's add this action to his repository in a few steps. I'll assume it's a Python project, but you can also add it to projects in different programming languages.
In the root folder of the repository, create the .github/workflows folders if it does not already exist. Now, within the workflows folder we are going to create a new file called test.yml with the following content:
# This workflow will install Python dependencies, run tests check the coveragename: Test with coverage
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
with:
python-version: 3.10
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install pytest pytest-cov
- name: Test with pytest
run: py.test -o junit_family=xunit2 --junitxml result.xml -v --ff --cov=<my_package> --cov-report=xml --cov-report=term <my_tests>
- name: Coverage Report
if: always()
uses: aGallea/[email protected]
with:
min-coverage-percentage: '100'
fail-under-coverage-percentage: 'true'
cobertura-path: ./coverage.xml
junit-path: ./result.xml
be sure to replace the and above with the name of your package and the corresponding tests folder.
That's all! If you open a new pull request to add this file, the action should fire automatically and you'll see the report:
Please note that since there are no changes to the package files (source files), there are no coverage details to present. The image above was taken from my pull request while adding test coverage action in one of my public repositories.
You have just completed the first action (double meaning) that will change your life and make you a better developer. Our generation is addicted to likes, claps, upvotes and geeks like us to show our professionalism with a 100% coverage report too. We would love to receive feedback, suggestions, and feature requests for this action that can improve your testing experience and motivation.
