A Brief Sprint Before the Marathon...

This post is the first of many as we complete our 12 month project.  Look out for more weekly updates starting on Fridays!

Last week, we dedicated our time to familiarizing ourselves with VistA documentation provided by OSEHRA and looking at some Mumps code.  This week we started looking at XINDEX (with respect to the utility’s results and what it is doing) in the context of a couple of specific applications -- Oncology and Problem Lists.  Note: these choices were random.  Our goal is to work in the context of a specific application before generalizing it.  One option we are considering is if there is interest in altering XINDEX to further refine its capabilities to look further into an application’s code (if you have any ideas, please feel free to push them our way).

A couple of observations from the code and documentation:

1)      Fileman can be considered the database layer for VistA.    It has a “Classic” version which provides command line user interface facilities for VistA applications in addition to database management.  This form appears to be the primary mode of usage for a lot of applications.

2)      However, Fileman also has a DBS (Database server) interface which has a well-defined “silent” interface and appears to be isolating the database layer quite well.  This does not mean all the applications are using this “good” interface…actually, we doubt they do, but at least we already have a database layer.

3)      The Fileman database layer is built on top of Mumps, but the structure is very similar to a relational database.  In fact, there is even some recommended relational database structure (SQLI) for an SQL interface.   (And if anyone has more information or recommendations for specific commercial SQL interface packages, please feel free to let us know).  We are under the impression that they provide read interface, however it was not clear if there were any write interface; this requires more investigation on our part.

A few thoughts on some applications briefly browsed:

They have a command line interface with a menu item structure: you choose a menu item from a selection list and you are presented either with another menu list or the item goes ahead and executes the action.  The documentation essentially describes the actions in the menu items.  It should be possible to list all the actions (essentially Mumps tags) from Option File (files are essentially SQL tables in VistA terminology).  That is something to keep in mind for dependency listing; we may want to specialize to tags instead of routines.

The applications that have a GUI (CPRS Problem List for example) appear to be better structured with defined APIs.  We think this is expected since those need at least a client and a server layer since more work is done on them.   There is an RPC broker and you can use it for any language to do calls to Mumps. 

Just as a heads up for what this project is looking for in the short term: towards the end of the month we will have to finalize documentation for lists of interfaces, common services, and all application layer modules...hence the 'sprint before the marathon.'  We will have another update to you next Friday with our XINDEX results and more – stay tuned!



SQL interaction

Ben Mehling's picture

I'll preface this by saying, our experience has been that you cannot rely on pure database interaction to enhance existing VistA-based applications. There's a lot of business logic entangled in the applications and not in the database model itself.  Much of this has been discussed over many years on the Hardhats list, another resource to consider.

As you say, SQLi provides hooks/metadata for SQL-like interaction with the database.  There are currently three (known) options for consuming the SQLi package...

1) Intersystems Cache has a FileMan->Cache Classes->ODBC endpoint. A routine is run (they've changed the name several times... I think it's FM2CLASS or FM2SQL) which uses the SQLi data to build Cache Classes (which by default can be projected as "RDBMS-like" structures via ODBC). This tool only works on Intersystems Cache and is included.


2) KBSystems KB_SQL product similarly extends the SQLi metadata into an "RDBMS-like" structure and provides some ODBC drivers as well as a native query tool. KB_SQL is agnostic of the underlying M-engine and works with both Cache and GT.M. It's installed at 40+ VA medical centers and the core technology is used in other products that support other M-based EHR systems.


3) FM Projection is an open source SQL Projection technology that uses SQLi's data to create the "RDBMS-like" structures.  It projects the FM data into the open source MySQL database server, leveraging the query engine/optimizations in that server. MySQL doesn't know the FM-backed tables are "special" in anyway, allowing for queries and joins across projected and non-projected tables. It is agnostic to the underlying M-engine and available under the AGPL/LGPL licenses.



Some more background on the relational model/SQL

Tom Munnecke's picture

Here are my comments about the relational/SQL discussion.  

VistA was carefully designed around a metadata model that was far richer than the relational model.  This was intentional, not an oversight.

Trying to "pigeon hole" medical information into densely packed 2 dimensional tables, even with sophisticated joins, is not a formula for success.  VIstA uses very high dimensional data structures which should be compared to Latent Semantic Indexing or Semantic Webs.  Also check out Semantic VistA

VistA could be considered on the leading edge of the NoSQL movement; Oracle has recently announced their radical departure to include a NoSQL product.  I haven't figured out Google's internal "secret sauce,"  but I do know that their large systems do not use SQL, but their Bigtable system uses  "a sparse, distributed multi-dimensional sorted map, sharing characteristics of both row-oriented and column-oriented databases... BigTable is not a relational database. It does not support joins nor does it support rich SQL-like queries. Each table is a multidimensional sparse map."  Nor does their App Engine support SQL or the relational database model.

I might also point out that Epic uses a database scheme very similar to VistA - in fact, they had very similar roots back to the MUMPS Users Groups meetings of 1970's.  Here is a paper describing Epic CEO Judith Faulkner's ideas from that time.

Clearly database technology has advanced since the design of the original metadata-driven approach used in VistA.  But I think that it is important to understand that this metadata model is at the core of the success of VistA.  Trying to refactor VistA without understanding this, or assuming that we can retrofit the complex linguistic relationships expressed in metadata into "flat" or densely packed relational tables is simply not going to work.

Another issue is performance.  The underlying distributed, cached b-tree storage technology is very efficient and finely tuned to the VistA architecture.  Even if folks were able to refactor to a relational database "in the small" for a single module, it would be guaranteed that this approach would scale up to VistA-scale production performance.



Ben Mehling's picture


Agreed.  In my nearly 10 year affiliation with VistA and its derivatives, this topic has been reviewed ad nauseum.

Any discussion of a 'conversion' to some sort of SQL like DBMS is a dead-end. If you count subfiles, VistA contains over 5000 files ("tables") and >40,000 fields. I cannot imagine the structures in an RDBMS.

On the other hand, exposting data via other interfaces, just like CareGraf's FMQL technology, is a useful pursuit. None of the technologies I point out are "conversions" or "replication" of data. They are all "projecting" the data. There is a lot of value in the data contained within VistA -- giving SQL report writers access* only extends the value of a VistA-based EHR.

* with the caveat that queries must be constructed carefully. That said, Conor (of CareGraf) has done sufficient work validating that alternative query tools are acceptable and accurate.


I second this

james mclemore's picture

Relational technology is excellent for analytics and queries.  On the other hand, relational technology almost always inhibits scaling.  AHLTA is a perfect example.  The patient record is only available in most cases if 24 hours notice is given.  Ironically, while the VA wishes to consolidate, AHLTA is moving to regionalize.

I hope any designs review the pitfalls of relational technology, because the Oracle rep will certainly be selling big-iron as the solution.

As we adopt web-based constraints such as http in order to meet the overarching goals of a complete patient centric health record, hopefully the pitfalls will become apparent.


Been venturing on homebased

shaun heatheridge's picture

Been venturing on homebased server recently. Starting up with a refurbished network ( http://www.spectra.com/ibm/ibm-system/46/17/index.htm ) That I've been wwanting to serve on multiple network. Since im deciding sticking on Vista, I would assume XINDEX would run properly as per this article says. Getting good on some opinions here though. Will definitely update once mine running. So far, my database loading time works fine.