VistA 'Onion' Diagram

Good Afternoon All;

My apologies for the delay on posting this to Julie; apparently my spam filter was not fond of your email.

Attached is the famous 'onion' diagram illustrating the layered growth of VistA over the course of its history.  This, combined with the VA's prioritization, has been shared with us to allow us to assist the community in prioritization of targets in the code convergence effort; in this we may target the core and iteratively expand our convergence efforts.

It is available in both power-point and pdf format.

Regards;

Matt

Upload

Comments

Tom Munnecke

A bit of history about the onion...

I sketched the original version of this diagram on a placemat at Coffee Dan's restaurant in Loma Linda, Ca. over dinner with George Timson in June, 1978.  It's an amazing experience to see this diagram still floating around, nearly 34 years later.

At the time, Ted O'Neill and Marty Johnson of the Computer Assisted Software Support Staff (CASS) were pulling together a team of programmers to work on a standardized, decentralized hospital information based on the MUMPS language.  Marty Johnson scoured the country to collect a team of really skilled programmers who had a lot of experience in health informatics, chosen for their skill and creativity over a range of specialties, but not necessarily their ability to "color inside the lines."

We planned to use minicomputers in hospitals, which at the time was a radical concept.  IBM mainframes dominated "real" computing, and were held in large computer centers doing mostly batch processing of administrative data.  The notion of using small, interactive computers to meet medical needs was seen by these centralists as 1) technically impossible, and 2) an existential threat to the centralized computing model.

The VA had a scattering of medically-related systems, all running in incompatible hardware, software, and administrative "stovepipes."  Most VA employees interacted with computers through forms that were keypunched, sent in to the Austin Data Processing Center, and never seen again.  Hospital barber shops would send in detailed forms to central office about the number of haircuts they performed, but there was no system for doctors to see lab information, clerks to find patients, or schedule appointments.  Information quality was atrocious: crosschecking the statistics generated by the centralized Patient Treatment File (PTF) and the Automated Management Information System ("Something is amiss with AMIS") were off by 600%. 

(I recall explaining to some central office folks about a way of extracting this information automatically - and reliably - from daily operations, they said, "But we rely on this data for our reporting to congress."  Being able to pick and choose data to suit their reporting needs was of great value to them.  This was my first taste of "beltway logic" - and the huge difference between technically correct and politically correct software development.  I formalized this later into Munnecke's Law of Software Productivity: that software development is inversely related to the square of the distance from VA Central Office.  (There were some exceptions to this rule, notably the imaging work done at Washington VA.)

I drew the original onion model to show how we might develop a common core infrastructure, driven by metadata contained in a data dictionary.  This would support a shared information environment upon which the applications would layer would sit.  The idea was to create a system that was "integrated" by the virtue of the fact that we had not "disintegrated" into pieces.  I didn't start with the pieces, then try to put them together, Humpty-Dumpty style, but rather tried to create a path of least resistance to a comprehensive, hospital-wide shared information system.  I just wanted to make it easier to doing things in an common manner, which was one of the goals of the kernel.

The original diagram had ANS MUMPS as the innermost layer.  Today, this could be called a Virtual Machine, as it was a simple interpreter based on 19 commands, 22 functions, one data type, on one language.  Porting this VM to another platform, ( coupled with the machine-specific function coding contstrained to Kernel ).  We can still see the effectiveness of this design decision today, as VistA can operate on either the proprietary Cache or the open source GTM interpreters.

One of the things I noticed was that everywhere I went in the VA org chart, people wanted to have things decentralized above them and centralized below them.  One of the critical inital design choices of the VistA effort was to decide where to anchor the design - was the center of the onion Central Office, a region, a hospital, or an individual service within a hospital?

DHCP focused on the individual hospital as this anchor point.  Given the technology of the time - "high speed" communication networks used 9600 baud modems; home computers were just appearing as hobbyist toys - I think that this was the appropriate choice.

When I saw features sprouting up at innappropriate layers of the onion diagram, I called them "warts on the onion."  The are quite visible on today's version, most noticably the Austin Automation Center poetically portrayed as an external amoeba-like lump.

This illustrates a key issue: the onion was designed to be a meta-level diagram of the connectivity and flow of information in a hospital, but now we have added a wart on the diagram that is based on the organization chart, not the data flow.  Instead of embedding the data flow into the inner layers of the diagram (I put it inside of the Kernel when I lead the kernel development team), it is now portrayed outside the onion, layered on top of COTS and Class III software.

This is a serious breach of conceptual integrity - we are moving down the slippery slope of "automating the organization chart" rather than meeting the needs clinical needs of the clinicians.  We can see this in the OSEHRA architecture design,  which goes into great detail about the organization chart objects, but seems to give scant attention to anything patient-related.

In looking back at the evolution of the VistA system from such simple origins, I am struck by a number of things:

1.  The endurance of this diagram after all these years.  Perhaps it is just bureaucratic inertia, but there is something charming about this diagram that pulls people to it.  Lesson learned: a successful, long term system needst to have a certain charm about it.... not too simple, not too complicated, but just enough so that folks can relate to it.

2.  The VA has failed to upgrade the infrastructure over the years, focusing on the outer layers of the onion... This can be seen in the confusing mish-mash of outer layers/warts as well as the ancient components of the inner layers that need a lot of modernization. 

3.  My biggest regret is naming the communications services subsystem "MailMan."  I was really designing a network infrastructure, with transaction processing, queuing, security, file formats, and social networking elements.  The actual delivery of email - what one sees with Outlook, for example - was just a minor portion of the design.  I wrote it as part of the March AFB/VA Loma Linda sharing effort, using an approach very similar to today's  HHS Direct Project

Next up: (if anyone reads this) Some thoughts on the future of the onion diagram.

But enought of the past 34 years.... what about the next decades?

mdoneill

A bushel full of slightly different onions?

Tom,

Always fascinating; you can be sure that plenty of people are reading what you post with great interest.

The path to any of the improvements or enhancements that we contemplate in your post and others have to be implemented on so many codebases, though - so many VistA variants. 

Isn't it worthwhile to focus on code convergence and achieve as much commonality as possible so that enhancements can be efficiently shared? The onion diagram is great, but if each instance is slightly (or greatly) different, then we are collectively on a never-ending treadmill of custom implementations for every new idea. 

Mike

 

Tom Munnecke

This has been my briar patch for 35 years...

A bit more of a timeline:  We had our first group meeting in Oklahoma City in December 1978, during which we met with folks from DoD, and Indian Health Service.  The whole idea was to create a government-wide health information system, and we laid out the ideas that became the Kernel we see today.  (The components developed separately for a few years; the name "kernel" appeared several years later; I was the first kernel development team leader in the 1981-1983 time frame).

This was a "rough consensus, running code" style of software development.  For example, here are my notes from the initial design of the security system http://munnecke.com/blog/?p=1345  (For the youngsters watching: there was a time before Powerpoint:  we used to use transparent overlays on projectors for presentations, and whip out our slide rule from its holster to use as a pointer to the screen.  Despite the crudeness of the handwriting http://munnecke.com/blog/wp-content/uploads/2011/12/1978-munnecke-initia... I managed to convey a design that drove the VistA, CHCS, and RPMS security systems. )

The whole idea was to build a common inner core to the onion, then use metadata if possible to adapt the outer layer functionalities.  (e.g. try to handle the DoD's Family Member Prefix code in the data dictionary, if possible.  If not, try to extend the capabilities of the kernel to handle this in a general way.  And if all else failed, do it in hard code in an outer layer of the onion.

As an aside, Judy Faulker, now CEO of Epic Systems, was dabbling with very similar approaches at the MUMPS Users Group meetings in the late 1970's.  She went off into her proprietary world, and I went off to the public domain world, so the technology transfer, if any, was pretty much one-way.  (I have been told by folks who know Epic and FileMan that they are amazingly similar, though.)

The fact that VistA, CHCS I, RPMS, and Kaiser Permanente are all based on architectural "first cousins" coming out of the 1970's MUMPS world (the language standard itself a federal IT project) should cause folks to dig a little deeper into just what made these things a success.  These systems were NEVER based on mainstream thinking, and the conventional wisdom ALWAYS was against this approach.  But the success of the systems is undeniable.  Collecting arbitrary non-medical IT experts and doing a consensus-based design will almost assuredly 1) blame MUMPS for the complexity of the system, and 2) present a Shiny New Thing approach that has a half life of a few years before a new Shiny New Thing comes into vogue.

I did the Loma Linda VA/March AFB interface in the early 1980's as a prototype to show how we could use the same kernel across the departments.  Rep. Sonny Montgomery had passed VA/DoD sharing legislation, and then Lt. Beth Teeple showed up in my office in Loma Linda asking to share computer systems.  One thing lead to another, and we very quickly had a working system with very happy users on both sides.  Here is a Nov 2011 video of Beth and me talking about this effort. 

This pleased Congress and GAO, but not the DoD.  They hired a consultant to study the March AFB project, who is now a senior level executive at an EHR company.  I happened to meet again just last month at HIMMS, and he confirmed what I had always suspected: that he had been hired by the DoD to make the system look bad.  He said that the system "looked pretty good to me," so he was in a difficult position.  Regardless, Montgomery pushed through legislation that one of the CHCS competitors would use an adaptation of the VA system (then called DHCP).  I moved to SAIC and worked on the CHCS effort, which we won with a bid of $1b compared to our nearest competitor at $1.7b.

My impression with CHCS was that DoD never really want it to be installed, largely due to a "not invented here" syndrome. I think that there was a lot of embarrasment about the cost and failure of the TRIMIS project, which spent $250m in prototyping incompatible departmental systems while DHCP was deploying 172 working, integrated hospital information systems.  My network design was jettisioned - it "exceeded requirements" only to be replaced by an entirely different half-vast approach that was incompatible with the VA.

The bottom line is, the forking between VA and DoD was a political/turf war problem, not a technical one.  Every decade or so, someone on the Hill gets excited about VA/DoD sharing, and huffs and puffs at the secretaries, who promise to finally do it right.  This leads to a doubling of the budgets and staffing for the (renamed) integration effort, but somehow gets bogged down in the execution, only to start another round of huffing and puffing on the hill.

The bottom line is that the VA/DoD sharing is a political problem, not a technical one.  We can draw onion diagrams and have technical convergence meetings forever, but until the political problems that have crippled the efforts for the past 40 years are solved, we're just spinning our wheels.

I'd be interested in Paul Tibbets' perspective on all of this... I know he was active on the DoD side through the TRIMIS, CHCS, and AHLTA progression.