OSEHRA Certification Tutorials Blog II

Stewardship of the code entrusted to its care is an important part of the mission of OSEHRA. This stewardship requires that OSEHRA take care that the code it accepts and certifies meets community standards for understandability, usability and correctness. Achieving this is not easy and the processes and procedures used can be difficult to apply. And yet, participation by a wide and diverse open source community is critical. Bridging the gap is crucial. The procedures must be understandable and community members need to be able to easily apply them when submitting, testing or evaluating code. To help publicize the code submission and testing process, and to provide a user friendly introduction, the Certification Work Group held a series of tutorial meetings. The recorded meetings instruct the community about the various facets of the OSEHRA testing process in brief (<25 minute) videos. These can be found at the following OSEHR webpage:


For the patient viewer, the full 45 minute source videos can be found at that link as well.


The Tutorial series is divided into two halves. The first half, presented in a previous post, focuses on the creation of certification tests using three OSEHRA compatible applications: Selenium for web pages; the “Roll and Scroll Recorder” (RASR) for scripted query/response interactions; and Sikuli for generic Graphical User Interface (GUI) interactions. The second half, presented below, covers the usage of the testing tools CMake and CTest to drive and report on testing results.



Tests are important, but only if they are used, monitored and maintained. OSEHRA recognizes that every user needs to verify correct operation of a code submission and to compare its behavior on their system against known baseline behavior. And yet, the tools used to test [M]umps code are far different than the tools used for Java. In fact, every language has its owned preferred set of development and testing tools that coders in that language know and understand, but that outsiders must learn. To bridge this gap and to make testing easy for everyone, OSEHRA mandates that developers use CMake and CTest as test wrappers. Users need only to learn the use of these two meta-test tools and they can execute any of the OSEHRA certified testing infrastructures. This insistence on a common testing environment puts an added burden on developers to learn and use the tools in generating their tests. Fortunately, CTest and CMake are can flexibly execute common test drivers using a simple syntax. To further ease the developer burden, OSEHRA provides a growing set of example tests for the various languages supported in the OSEHRA Technical Journal (OTJ).


In the fourth, fifth and sixth tutorials, we talk about CMake and CTest; the mechanics of making an OTJ submission with testing code; and the mechanics of setting up your own testing infrastructure. The tutorials present the history, and rationale behind the use of the CMake family as the driver for the OSEHRA testing harness; give practical examples of how to use the tools to generate and execute tests; show how to make a submission to the OTJ that incorporates documentation, code and tests; and, finally, how to setup a ready-to-test environment using some simple OSEHRA tools. In doing this, the tutorials also highlight the virtues of automated testing and why it becomes especially important for a project with the size and complexity of an EHR.  


More specifically, the CMake/CTest tutorial (fourth in the series) specifically presents the command structure of the OSEHRA VistA repository, showing the CMake code that sets up the commands for CTest to execute, and illustrating the multi-level folder structure we used to keep the command (CMakeLists.txt) files short and understandable.  In the demonstration portion of the tutorial we create a test specification in a CMakeLists.txt file and demonstrate the resulting test execution.


The fifth tutorial creates a submission for the OSEHRA Technical Journal (OTJ) by presenting the materials that need to be prepared for a complete submission.  Special attention is paid to the following points:

  1. The Technical Article can only be submitted in PDF form, other documentation may be in other document formats  such as .doc/.docx or .txt

  2. Source code may be submitted in an archive file (.zip/.tar.gz) or as a Git repository.  If a repository address is given, the OTJ will clone and archive the master branch for download.

After the preparation of materials, we illustrate the process of submitting an article by  the test code using the M-Tools software. The submission of the M-Tools software from the tutorial can be found at in the OTJ at http://hdl.handle.net/10909/106


Finally, the sixth tutorial highlights an OSEHRA provided capability for automatically “refreshing” a VistA instance.  Leveraging the testing infrastructure, an OSEHRA MUMPS source tree is imported into a clean VistA instance. To ensure a clean start, the scripts overwrite the MUMPS database prior to the import. Due to this overwrite of the MUMPS database file(s), this "refresh" capability should NOT be used in a live or production environment. Options allow the addition of a minimal non-production VistA instance containing users (System Manager,Doctor, Nurse, and Clerk), fictional patients, and even a few clinics on completion.  This is a critically important capability for developers and for testing because it allows testing the return to a specific known state of the code. It is used to rebuild the testing instances for the Nightly Dashboards and to populate the MUMPS database during the Vagrant VM setup.



Misleading information re Refresh

DAVID Whitten's picture


You know that I value all the work mentioned in this posting. A lot of hard work has been done to make development and sharing easier for developers and maintainers of VistA code.

I want to point out that your discussion of "refreshing" can be misleading to an adopter of VistA. You suggest that to take an instance (which may be running in a medical facility) and refreshing the code and data for a clean start is a good thing. 

This would be DISASTROUS for anyone doing so in a live production system. 

Use of such a refrested instance for testing and for developing new VistA functionality is fabulous, and can really help make better and more robust code. Using the "refresh" methodology should NEVER be used in any populated database that is expected to be used to deliver healthcare.

Updating a live system should be done using KIDS and the PATCH module. It should not happen using a any source code repository.

I know that these facts are well understood by OSEHRA. I just want to make sure new adopters don't get confused by this blog posting

David Whitten



Thanks David, That is a topic

Joseph Snyder's picture

Thanks David,

That is a topic that I think we touch upon during the meeting and tutorial, but it is best to put that information here as well.

I've edited the paragraph to have a bolded statement with that message.  Thanks again for the always on-point comments.

- Joe