Recently I've started testing a Haskell application and a question I find unanswered (or at least very poorly documented) is how to produce coverage reports for binaries ?
Understanding HPC & cabal
hpc is the Haskell code coverage tool. It produces the following files:
- .mix - module index file, contains information about tick boxes - their type and location in the source code;
- .tix - tick index file aka coverage report;
- .pix - program index file, used only by
The invocation to
hpc report needs to know where to find the .mix files in order
to be able to translate the coverage information back to source and it needs to
know the location (full path or relative from pwd) to the tix file we want to
cabal is the package management tool for Haskell. Among other thing it can be used
to build your code, execute the test suite and produce the coverage report for you.
cabal build will produce module information in
cabal test will store coverage information in
A particular thing about Haskell is that you can only test code which can be
imported, e.g. it is a library module. You can't test (via Hspec or Hunit) code which
lives inside a file that produces a binary (e.g. Main.hs). However you can still
execute these binaries (e.g. invoke them from the shell) and they will produce a
coverage report in the current directory (e.g. main.tix).
Putting everything together
cabal testbuild the project and execute your unit tests. This will create the necessary .mix files (including ones for binaries) and .tix files coming from unit testing;
- Invoke your binaries passing appropriate data and examining the results (e.g. compare the output to a known value). A simple shell or Python script could do the job;
- Copy the
Produce coverage report with hpc:
hpc markup --hpcdir=dist/hpc/vanilla/mix/lib --hpcdir=dist/hpc/vanilla/mix/binary dist/hpc/vanilla/tix/binary/binary.tix
Convert the coverage report to JSON and send it to Coveralls.io:
cabal install hpc-coveralls ~/.cabal/bin/hpc-coveralls --display-report tests binary
Check out the haskell-rpm repository
for an example. See job #45 where there is now
coverage for the
rpm2json.hs files, producing binary executables.
Also notice that in
parseRPMC is now covered, while it was not covered in the
previous job #42!
script: - ~/.cabal/bin/hlint . - cabal install --dependencies-only --enable-tests - cabal configure --enable-tests --enable-coverage --ghc-option=-DTEST - cabal build - cabal test --show-details=always # tests to produce coverage for binaries - wget https://s3.amazonaws.com/atodorov/rpms/macbook/el7/x86_64/efivar-0.14-1.el7.x86_64.rpm - ./tests/test_binaries.sh ./efivar-0.14-1.el7.x86_64.rpm # move .tix files in appropriate directories - mkdir ./dist/hpc/vanilla/tix/inspect/ ./dist/hpc/vanilla/tix/unrpm/ ./dist/hpc/vanilla/tix/rpm2json/ - mv inspect.tix ./dist/hpc/vanilla/tix/inspect/ - mv rpm2json.tix ./dist/hpc/vanilla/tix/rpm2json/ - mv unrpm.tix ./dist/hpc/vanilla/tix/unrpm/ after_success: - cabal install hpc-coveralls - ~/.cabal/bin/hpc-coveralls --display-report tests inspect rpm2json unrpm
Thanks for reading and happy testing!