OSEHRA Certification: Code coverage capabilities keep growing.

Recently, the OSEHRA Certification Workgroup (CWG) has been busy with the certification of two excellent submissions to the OSEHRA Technical Journal (OTJ).  Along with being generally valuable in and of themselves, these two submissions also played a pioneering role for OSEHRA: they were the first submissions to undergo certification using the Certification Standards V2 document, which enhances our process to give out “Certification Levels” based upon 8 dimensions of Open Source quality. While most of the categories should be familiar to those of you who have participated in previous certifications, one notable addition to the process is the formalization of the regression testing. The “Regression Testing” category has been in place to ensure that the current tests continue to work and that new submissions come with tests.  The new guidance specifically calls out code coverage levels which must be met to achieve Level 3 (50% code coverage) or Level 4 (90% code coverage).  This new requirement raises the profile of our coverage capability and ensures that OSEHRA will need to be continually working to maintain and enhance the coverage capabilities that are available through the Testing Harness. Coverage can currently be calculated for two languages which are widely used in the OSEHRA community and planned enhancements will add more languages.


The first and most important addition to coverage for the OSEHRA project is the M or MUMPS language.  This ability, first available in CMake version 2.8.9, has become an integral part of the OSEHRA certification process.  The coverage is calculated using the "VIEW TRACE:.." capability of GT.M or the %SYS.MONLBL on Intersystems Cache, both of which write out coverage results to the file system in a format so they can be read and parsed by CTest.

The M[UMPS] coverage ability is essential to OSEHRA certification operations. As such, every night the OSEHRA automated testing infrastructure runs a MUMPS coverage suite and submits the coverage results to the OSEHRA Open Source Dashboard. This same coverage reporter was most recently used in the certification of the M-Unit Test framework.  Notably, the M-Unit Test submission had excellent coverage and received a Level 4 certification -- the first to reach that level.  This required that the tests that accompanying the submission covered greater than 90% of the supplied code.  A snippet of the coverage report is shown above and the full coverage report can be found here.


OSEHRA is increasingly accepting code submissions in languages other than Mumps, including a number of submissions in Java. These incoming Java projects now have two options for coverage tools that work well with the OSEHRA Testing harness: Cobertua and Jacoco, each described below.  



Cobertura is a flexible and popular coverage reporting format that has been used for CMake Python code tests. OSEHRA has also embraced Cobertura as a standard for reporting Java coverage allowing us to utilize any coverage generator programs which is able to write out a Cobertura compliant XML file



OSEHRA has also added an entirely new Java coverage tool, Jacoco, to the mix.  Jacoco is an emerging standard that is still under development by the EclEmma team and is an upgrade to their previous coverage tool EMMA. This tool and its coverage parsing served, in conjunction with the JAVA Build Management tool Maven, as the basis for the August 2014 certification of the OpenInfoButton submission. The OpenInfoButton submission achieved Level 3 certification by providing tests covering greater than 50% of all code submitted. An example of the Jacoco coverage on the OSEHRA dashboard can be found here and here.

Future Work


An ever widening range of languages and tools are being brought to bear in the Health IT (EHR) space and more and more of them are finding their way into OSEHRA submissions.  Thus, the push for OSEHRA to keep adding more coverage options continues. OSEHRA is currently vetting tools and generating the parsers for three additional languages:

Delphi via Delphi-code-coverage


A number of GUI front ends connect to the VistA EHR, most notably the VA authored CPRS interface. These GUIs have long been used as a provider portal into the EHR, allowing doctors and nurses to enter information in a graphical form rather than executing calls on a command line. Written in Delphi Pascal, they offer special challenges for OSEHRA certification. OSEHRA has begun to generate the files necessary to parse the output of the Delphi-code-coverage tool and report the results on the OSEHRA Dashboard.  An example submission of the code coverage for a few Delphi/Pascal files can already be found on the OSEHRA Dashboard.  

Javascript via Blanket.js


Javascript is also gaining an expanding role in the OSEHRA toolbox.  Being able to calculate coverage for Javascript will fill a need both internal to OSEHRA, regarding the testing and potential certification of the VIVIAN tool, and to the community, with the potential influx of EWD.js-based GUIs for VistA and the other EHRs.  The OSEHRA harness does not currently directly process the output of the Blanket.js tool at this point, but instead relies on post processing in each project to write to a readable XML format. OSEHRA will be adding direct Blanket.js support into CTest using the test runner library Mocha.js. This will simplify the coverage generation process for Javascript development.

Ruby via SimpleCov

Finally, SimpleCov is a Ruby coverage tool. The OSEHRA team is in the preliminary stages of generating a Ruby testing platform and are evaluating the effort that would be involved in using SimpleCov. Expect more news in the near future!