VistA-based open source web services

 

I didn’t have the privilege of attending Rob’s AWG presentation but I did watch the recording and catch up on the blog postings – yes, including the openMDWS cookbook.

After digesting the excellent openMDWS discussion, along with an equally excellent Thanksgiving meal, I’m finding that the key architectural goals of openMDWS and MDWS aren’t so different after all. That’s worth exploring…

The openMDWS architecture incorporates core re-usable functions, full variable scoping, no leakage of VistA variables and wrapping code in a function. It’s very well explained in the openMDWS cookbook blog post. I’m all for it – sounds great. And familiar!

Since the inception of the MDWS project on OSEHRA we have worked hard to follow precisely the same architectural principles. From the get-go, the MDWS team specifically chose to use the refactored scheduling codebase for the same reasons Rob has articulated. The refactored scheduling codebase exposes a mumps API of re-usable functions with full variable scoping, etc. (Kudos goes to the EHR refactoring group for taking it one step further by abstracting a data access layer and documenting the API.) Once the mumps API is exposed you can wrap openMDWS, MDWS, jMeadows, closedMDWS, and whatever you like around it.

So really there aren’t core architectural differences in the design principles of openMDWS and MDWS.

They do differ in implementation, however. MDWS uses the RPC broker as its intermediary pipeline to connect and query a VistA instance, which by its very nature is stateful. So to address the problem and quickly summarize a lot of my thinking I’ll pose the following question: Is it easier to make an existing stateless service stateful, or to make an existing stateful service stateless? Now mix in a little bit of the RPC broker implementation details to the equation. My opinion: It’s easier to make an existing stateful service stateless. Which, as you can imagine, is what we did for the MDWS scheduling effort.

MDWS also differs by its use of the .NET framework to create consumable services (SOAP, REST, WebSockets, and any other imaginative endpoint you can think of). For the MDWS scheduling effort we aren’t focused entirely on exposing it as a SOAP service.

A .NET implementation of MDWS won’t work for individuals who don’t run a Microsoft OS or an open source .NET development framework. That’s a fact, but it’s one that, given enough community interest in converging on a common set of web services, we might address by moving off the .NET platform.

Scalability has been described as a problem yet it is unknown to me in the VA using MDWS or jMeadows.

MDWS can feasibly read and write to any portion of VistA. I say feasibly because the technical limitation to read or write from VistA is not a MDWS limitation but a limitation of the VistA system itself. The VistA scheduling package is an example that depicts the VA’s inability to modernize due to numerous factors such as technical debt. VistA’s scheduling technical debt can be described as a lack of loosely coupled routines separating business logic and data access from the traditional character user interface. It can also be described as a lack of a mumps API to allow other systems or software (MDWS, openMDWS) to interoperate. Therefore to successfully read and write to VistA from an outside system or software, a mumps API must be exposed, following the previously described architectural principles such as re-usable functions and full variable scoping. VA is addressing a portions of these issues through the OSEHRA EHR refactoring services contract to RGI (now PWC).

MDWS can be enhanced by anyone. I recommend developers’ fork the OSEHRA MDWS projects on GitHub. Commit their changes, document the enhancement and issue a GitHub pull request. The MDWS team will gladly review the proposed changes.

And lastly, thinking about community convergence on common web services… it may not make sense for VA to adopt M-based services when they could more easily use web services provided by their InterSystems licenses. And for folks outside VA, who may be running GT.M, it’s not clear that building M-based services offers any special advantage. A set of web services developed on a non-M platform seems to allow the broadest adoption by users inside and outside VA. If we’re all interested in enabling the largest set of application developers to develop VistA-based software, isn’t that all that matters?

 

like0

Comments

Chris Thanks for this well

Rob Tweed's picture

Chris

Thanks for this well written blog posting.

I agree that openMDWS and MDWS are and should be about doing compatible things.  However, openMDWS is about keeping it open and permitting the MDWS principles to be applied to the wider open source community of VistA users/developers outside the VA who don't use Cache and/or won't or can't use Microsoft or other proprietary technologies.  

openMDWS is also about making best use of the technology that VistA already sits on and uses.  That technology turns out to not only just as capable as any of the other technologies used to provide web services etc, but benefits from the in-built strengths of the Mumps technology - namely massive scalability and performance.  

My blog articles (http://robtweed.wordpress.com) were written to debunk a great many myths and a lot of disinformation about Mumps that I've watched grow and develop over the last couple of decades, and open the eyes of the new generation of developers who have to work with and co-exist with this myserious Mumps technology, to what it really can do and how it is actually ready and able to play in the modern networked world.  Contrary to what I've discovered are most people's belief, it can do such things remarkably easily and extremely efficiently.

The web architecture that is available natively for Cache and GT.M has almost no moving parts - an important issue when maintaining and supporting a production platform, and also when debugging applications.  Why use a byzantine tower of technologies when you can pretty much plug straight in without any additional technology at all, and when the outward appearance is a standard web service?  I'd liken it to the marvellously unecessary contraptions drawn by Rube Goldberg in the US, or Heath Robinson in the UK which set out to achieve simple tasks in the most convoluted ways possible.

It's not at all clear to me, therefore, what the benefit of your suggestion of developing a set of web services on a non-M platform is when:

- like it or not, VistA is written in M code and runs on an M platform, so the technology and the capabilities are sat there anyway;

- it's not necessary in the first place;

- all you're doing is exposing back-end M code via a web service request/response (ie we're not talking about consumption here);

- as I hope my series of articles on openMDWS demonstrated, aside from the Mumps development that has to be done *anyway* by a VistA expert to encapsulate the required logic as a service, pretty much everything else is handled automatically by EWD's openMDWS technology to expose that developer's work as a web service.

The technology used to implement web services won't have the slightest effect on their external use or uptake - the whole idea of web services is to hide and make irrelevant the underlying technical architecture of the services.  What matters is that those services are architected in the best, most efficient, most productive and most maintainable way.

I'd also take issue with your suggestion that the InterSystems web service capability is any easier for the VA: it requires object wrappers/mappings to be created around the VistA M code.  Furthermore, this would render those services unusable by the non-Cache VistA development community.

If the benefits of OSEHRA and the open-sourcing of VistA are to be a two-way thing between the VA and the outside open source community, then anything built around the proprietary features of Cache just isn't going to work or be acceptable to the open source community.

Anyway, I think it's good that we're having this debate, and I'm pleased it didn't prevent your enjoyment of your Thanksgiving Dinner! :-)

Rob

 

 

like0

Van Come Lately

Van Curtis's picture

This and Rob's original post were pointed out to be again yesterday, and Joel and Joe and Chris and Imram and Rob have already said much of what needs to be said. I'd just like to add a few cents worth:

1. As has been mentioned, due to the nature of MDWS's inception we had to build it without *any* mods to VistA. No patches, no additional keys or menu options, nothing. Thus the constraints of piggy-backing the RPC listener used by CPRS. Also why it was developed on top of VistA rather than from within. This may not be an issue for other VistA deployments, and it is my hope that folks like Rob will help trailblaze a new externally facing API from VistA that allows for cleaner read/writes. MDWS does a lot of things it shouldn't need to do because of what is available *in production, in the VA.* As this changes, we jump on it and reveal new APIs (such as those from AViVA/HMP) as they become available. (This is also points to reasons why there are "many moving parts" with MDWS -- we don't have control over the whole stack or the resources to develop and release a monolithic solution. We work with other efforts.)

2. The other really big thing about MDWS is that it can connect to all the VistAs in the VA simultaneously, not just one, providing client applications with patient-centric information, not just facility centric information. I've probably missed in Rob's EWD writings how web services running out of one VistA are used to service multiple VistA systems at once, but it seems to me that sometimes having a server that sits on multiple data sources is a desirable thing.

3. Speaking of multiple data sources, MDWS doesn't just talk to VistA, it talks to other types of data sources, including the Oracle-based HDR where it gets some of the information it supplies to MyHealtheVet, for instance. And the client application doesn't have to know where that data comes from.

4. On the subject of open source, I'm as big a proponent as any. The choice we had to make (besides which platform was easier for us to develop web services) was, "which platform provides the greatest opportunities for reuse and collaboration *in the VA*." The answer to that for me was .NET because the large number of windows desktop machines, the ability to easily stand up a development instance of MDWS, point it at various test data sources, and completely screw up your app or your copy of MDWS without hurting anybody else. Indeed, many of MDWS's early clients and collaborators were people with experience in Delphi or VB, not Java or MUMPS. So we would write business delegates for them that they could just drop into their applications. For the enterprise development folks that are full time developers, we figured that Java developers could more easily make the jump to C# than VB and Delphi folks jumping to Java. So as weird as it seems, .NET was the "open source" solution within the confines of the VA.

I have nothing bad to say about MUMPS or EWD. I'm honored by Rob's nod to MDWS in naming openMDWS (though it's a bit misleading), and that he should choose us as the game to beat in town in his writings. There are other folks who have been decrying MDWS for years and forecasting its demise. But the fact is that it has survived and is in use against 150 production VistAs (or how many ever there are these days) every day and has been for years. Until Rob makes a billion dollars selling EWD back to the VA (and I hope he does) MDWS will continue to be here in the meantime. If folks find inspiration (or faults) in MDWS that help them to deliver better healthcare solutions for everyone, I'm all for it. My motto has always been, "develop it and give it away as fast as I can."

Thanks for reading,

van.

 

like0

A billion? I wish!! :-)

Rob Tweed's picture

Thanks for the very informative follow-up, Van.  That's all useful background to the history of MDWS.

Just to answer your question on accessing across VistA systems via EWD.  At the time of my writing my postings about openMDWS, I hadn't attempted to address this.  However, due to requirements of a certain competition, EWD / openMDWS now includes a peer-to-peer mechanism that allows any VistA system to securely access any other.  I've used the same HMAC-SHA-based mechanism for signing HTTP(S) requests that Amazon Web Services (AWS)uses for their cloud services (eg S3, SimpleDB).  Just as Amazon AWS shares a secret key with each user, so each VistA system needs to know the secret key of any other VistA system it needs to communicate with.  This ensures that a door isn't left wide open for everyone under the sun to have HTTP/web service access to a VistA system, and it means that a VistA system could be made accessible via web services across the Internet, just as happens with Amazon AWS's cloud services.  Whether the VA wish to make use of this, I don't know, but it should make a lot of sense for non-VA users of VistA.  It certainly makes for an easy to set up and easy-to-administer cross-site communication mechanism, without the need for some centrally-managed server on a hub-and-spoke basis.  The key issue is to make sure that the secret keys are communicated between VistA sites in a secure, out=of-band way.

As to the name openMDWS: someone had to grab it: it was too good a name to miss! :-)

Hope this helps.

 

like0

Why a middle tier should be stand alone

Joe Gillon's picture

Why a stand-alone middle tier?

Joe Gillon's picture

From http://vistacowboy.com/?q=content/belly-wash

“Cut out the middle man”.  That's a good thing, right? Well, not

always. In court, you might find a lawyer useful, or a realtor when
buying or selling a house, or any number of other examples. Yes,
there's a cost, but in the long run you come out ahead. The same goes
for a stand-alone middle tier. Sure, you certainly save some network
traffic when your client apps go straight to VistA instead of going to
VistA through a middle man like MDWS, and it certainly sounds
economical to have all your business logic in one place and to “build
on legacy”, but at what cost?     Well, for starters, your business
logic is inside VistA. In M code. Here's an example of what that looks
like:
 
TOT	N PRCA,PRCB,PRCG,PRCF,PRCH,PRCJ,PRCK
	S T="" S:$D(^PRCS(410,N(1),4)) T=^(4) S X=^(0),Z=$P(X,"^",2),T(0)
=$P(T,"^",5),T(1)=$J($P(T,"^",8),0,2),T(3)=$P(T,"^",14),T=$J($P(T,"^"
,3),0,2),PRCA=$G(^(4)),PRCB=$G(^(7)),PRCH="*^*"
	I $P($G(^PRCS(410,N(1),1)),"^",2)=9999999 S PRCH=""
	S PRCF=$G(^PRCS(410,N(1),0)),PRCG=$P(PRCF,"^",2),PRCK=$P(PRCF,"^"
),PRCF=$P(PRCF,"^",4),PRCK=$P(PRCK,"-",2)_$P(PRCK,"-",3)_$P(PRCK,"-",
5) PRCG="A",PRCF=1 S:$P(PRCB,"^",6)]"" PRCS("C")=PRCS("C")-T(1),$P(PR
CH,"^")="" S:$P(PRCA,"^",10)]"" PRCS("O")=PRCS("O")-T,$P(PRCH,"^",2)=
"" Q:$D(C1)  G WRT
	I PRCG="O" S:$P(PRCB,"^",6)]"" PRCS("C")=PRCS("C")-T(1),$P(PRCH,"
^")="" S:$P(PRCA,"^",10)]"" PRCS("O")=PRCS("O")-T,$P(PRCH,"^",2)=""
	I PRCG="C" S PRCH="",PRCS("C")=PRCS("C")+T(1),PRCS("O")=PRCS("O")
+T
	I PRCG="A" S PRCH="",PRCS("C")=PRCS("C")-T(1) S:T(3)'="Y" PRCS("O
")=PRCS("O")-T
	I PRCG="CA" S PRCH="#^#"
	S PRCJ=$P($G(^PRCS(410,N(1),4)),"^",5)
	I PRCH'["#",PRCJ'="" S PRCJ=$P(^PRCS(410,N(1),0),"-")_"-"_PRCJ,PR
CJ=$O(^PRC(442,"B",PRCJ,0)) I +PRCJ'=0,$P($G(^PRC(442,PRCJ,0)),"^",2)
=25 S X=$G(^(7)) D
	S:PRCG'="A" PRCH="@" S:$P($G(X),"^",2)=40!($P($G(X),"^",2)=41) PR
CH="^" S:$P($G(X),"^",2)=50!($P($G(X),"^",2)=51) PRCH="&"
	S T=$P($$FP^PRCH0A(+PRCJ),U,2),$P(PRCH,"^",2)="R"
	QUIT 
 
 
Nice, huh? Would it surprise you to learn that many developers 
describe this as hieroglyphics? From just a cursory glance try to 
guess what it does. Now here's the same logic in a contemporary 
language, in this case, Python:
<pre>
    if line_item['transaction_type'] == 'CEI':
        balances['control_point'] = current_balances['control_point'] 
        + transaction_amount
        balances['unobligated'] = current_balances['unobligated'] 
        + obligated_amount
    elif line_item['transaction_type'] == 'ADJ':
        balances['control_point'] = current_balances['control_point']
        - transaction_amount
        if line_item['adjust_control_point_balance_only'] != 'Y':
            balances['unobligated'] = current_balances['unobligated']
            - obligated_amount
    elif line_item['transaction_type'] == 'OBL':
        balances['control_point'] = current_balances['control_point']
        - transaction_amount
        balances['unobligated'] = current_balances['unobligated']
        - obligated_amount
    elif line_item['transaction_type'] == 'CAN':
        pass
    else:
        raise VistaException('Invalid transaction type: '
        + line_item['transaction_type'])
 
You don't have to be a Pythonista to see this has something to do with
calculating balances for control point line items. Which would you
rather  write? Or worse, which would you rather support? Imagine how
much easier it is to attract Pythonistas than it is to attract
MUMPSers. In fact, you can get Pythonistas "off the rack" but
MUMPSers have to be home-grown.
 
But that's only a minor problem with having your business logic inside
VistA. There are bigger issues. Are you going to be in VistA forever?
Isn't it possible you may have to or want to migrate out of VistA
someday? And if so, is that day really so far off? Before answering
think of the roll'n'scroll user interface; think of the mess that is
the labs package, or scheduling, or &lt;insert your favorite VistA mess&gt;;
think of how impossible it is to query VistA. VistA has been around a
very long time. It has warts. Boy, does it have warts. The VA has
already tried to migrate out of it with the failed HealtheVet
initiative. Just because they failed doesn't mean it wasn't a good
idea. So if you might want to get out of M/VistA, isn't it an odd
notion to be adding to it? To be writing even more M code? Doesn't it
make more sense to get all that logic out of M and into something
modern? Imagine if that were already the case. VistA then would be
nothing more than a data repository, like Oracle or SQL Server. You
might then think, hey, do we really need non-medical data like
inventory, financial, employee time, etc. in a hierarchical database?
The major justification for a hierarchical database is speed, which is
very important at point of care but less so in these other use cases.
And because your logic is outside VistA in a stand-alone middle tier
it suddenly becomes pretty easy to replace your hierarchical data
repository with an SQL one, that is, with one you can actually query.
So a major cost of putting your middle tier inside VistA is the
retardation of any migration effort. And conversely, a major benefit
of a stand-alone middle tier is the advancement of any migration
effort.
 
But wait, that's not all. Business logic inside VistA can be applied
only to VistA. Suppose you need to get data from some SQL or XML
source as well as from VistA? You know, like Google Health or
Microsoft HealthVault. Uh oh, those business rules are locked inside
VistA. Now you have to extract them in order to apply them to other
data sources. Just as you could have done in the first place with a
stand-alone middle tier. You also have to merge these different data
sources for consumption by the user. To clarify, suppose you have
multiple types of data sources that have allergy data. You don't want
your user to go to one source, get the allergies, go to another
source, get the allergies, etc. Your user should just say, “show me
this patient's allergies” and your software should know how to connect
to each, authenticate and authorize at each, get the data from each,
all simultaneously, figure out the different field names and then
blend these disparate results together into something like a reverse
chronological list. You know, just the way MDWS does, something it can
do because it is a stand-alone middle tier. And because it is a middle
tier, its client apps don't have to reinvent these wheels every time.
The client apps don't need to know how each data source works, they
don't need to do the multi-threading for the parallel fetching. That
is all encapsulated in the middle tier and is a one-time cost to the
organization. And if that logic ever changes, if new sources are added
or whatever, the new code only has to be done once and the client apps
never know it happened.
 
And finally, what about two or three years from now, when some
combination of hardware and software comes along and produces the most
awesome, nearly perfect database imaginable? With a stand-alone middle
tier you're at least on the path that leads to that database. With a
middle tier inside VistA you first have to extract all that business
logic into, wait for it... a stand-alone middle tier.
 
like0