The US Department of Veteran’s Affairs (VA) has embarked on an initiative known as VistA Evolution, the goal of which is to create a new generation of their Electronic Healthcare Record (EHR) that will be known as VistA4.
The aims of this initiative are entirely laudable: to modernize the aging, but much loved and very successful, VistA EHR, improve its functionality in areas such as scheduling and make it more accessible for others to build upon using modern tools. The VA also recognizes the importance of making VistA, or rather key parts of its functionality, accessible from the growing range of mobile devices.
The VA has begun awarding a number of very high-value contracts under the umbrella of the VistA Evolution initiative (eg to ASM Research/Accenture), but in my opinion there are problems looming on the horizon. From what I understand about the direction that these projects are taking (with encouragement, it seems, from within the VA), there’s a real risk that we’ll see a repeat of previous attempts to modernize VistA, the result of which was very expensive failure with essentially nothing to show for it. The losers, if this happens, are not only US tax-payers: it’s the Veterans whose future welfare depends on VistA4 being a success.
So, what’s wrong with the VistA Evolution picture that is beginning to emerge, and what do I suggest as an alternative approach that will learn from and avoid past mistakes? In order to answer these question, we first need a bit more background.
At the heart of the issue is VistA. VistA is the VA’s EHR that supports and manages every aspect of the healthcare provided to US Veterans. The history of its development is steeped in controversy because VistA arose through the largely clandestine activities of a group of VA programmers and clinicians who worked together as what became known as the Underground Railroad. The VA’s strategic approach at the time was to create a centrally-managed, mainframe-based system. However, whilst the official VA efforts cost a great deal and achieved little, the unofficial efforts of the Underground Railroad, based on a decentralized network of mini-computers, proved highly successful, despite efforts by the VA’s IT management to shut down their work. Eventually, in 1982, the VA capitulated and legitimised their work, naming the product the Decentralized Hospital Computer Program (DHCP). In 1996 the name was changed to Veterans information system technology Architecture, abbreviated to VistA.
A critically important factor is the technology on which VistA is based. The Massachusetts General Hospital Utility Multi-Programming System, otherwise known as Mumps or often abbreviated to M, was first developed around 1966-7 and was specifically designed with the requirements of healthcare in mind. It is testimony to the design and thinking behind the Mumps technology that even to this day it dominates the healthcare sector, worldwide. Almost all of the most successful EHRs are based on Mumps or a modern derivative. The problem with the Mumps technology is that is almost unknown in the wider mainstream of the IT industry, and the few that have actually heard of it generally have a very poor understanding of it. This is due to a number of reasons:
In my opinion, the success of the Mumps technology within the healthcare sector is entirely due to these unique NoSQL characteristics of the Mumps database. Unfortunately, within the IT industry (and, sadly, within the Mumps community itself), Mumps is considered to be first and foremost a language rather than a database. As a result of the old-fashioned nature of and perceived deficiencies in its language, it gets dismissed as an obsolete technology. This is unfortunate, because if it was viewed as first and foremost a database, a very different conclusion would be reached. The IT mainstream therefore throw a uniquely-capable database baby out with the Mumps language bathwater.
EWD.js is an Apache 2-licensed, Open Source framework for building modern browser-based applications. It is primarily designed to integrate with Mumps databases, and is ideal for modernizing legacy Mumps applications. Key factors are:
The diagram below summarizes how EWD.js integrates with VistA. The three boxes colored red indicate the areas where developers are required. Everything else either just works or exists already:
VistA is highly regarded by its users – its functionality, in terms of both breadth and depth, equals and in many areas exceeds that of even the most successful and well-known commercial EHRs. However, that VistA needs to be modernized is beyond question, and in two key ways:
These two deficiencies are, in a nutshell, what VistA Evolution aims to address.
VistA Evolution isn’t the first time at attempt has been made to modernize VistA.
In 2001, the VA began an initiative known as HealtheVet, the aim of which was broadly similar to today’s VistA Evolution initiative. Technically, the intention of HealtheVet was to replace the Mumps database of VistA with the Oracle relational database, and to implement an overarching technical architecture that was based around the Java programming language and a set of Java-based services that is known collectively as Java 2 Platform, Enterprise Edition (J2EE), now known as Java EE.
In 2008, the HealtheVet initiative was abandoned after the expenditure of some $600 million. At that time it was estimated that the project would take until 2018 to complete, at a cost of $11 billion. One of the outcomes of this hugely expensive and highly politically embarrassing failure was the decision by Roger Baker, VA CIO (2009-13) to make a significant strategic change in direction and to move the VistA EHR into Open Source.
His rationale, ratified by Congress, was the recognition that VistA was functionally adequate and it made no sense to replace it outright, and furthermore to recognize that there was a growing community of developers outside the VA who had detailed technical knowledge of VistA and were already embarking on modernization exercises, from whom the VA potentially stood to benefit. As part of this new approach, Baker established Open Source Electronic Health Record Alliance (OSEHRA), the custodial agent responsible for managing and coordinating the open source version of VistA.
The official reasons given by the US Government Accounting Office (GAO) for the failure of the HealtheVet initiative focused on management deficiencies, particularly insufficient project management expertise and adequate milestones. What tends to be overlooked is the technical side of the project, in particular the architectural complexity of the Java EE platform on which it was based. This can be clearly seen in one of the official presentation slides that summarizes the planned HealtheVet architecture. This slide was created by one of the project’s technical team at the time:
One interesting thing is that the complexity that is plain to see in this diagram is not actually unusual in a Java EE architecture. The other interesting thing is that the Java EE platform has been very successfully implemented elsewhere and underpins many huge enterprise-scale IT operations around the world. Perhaps the conclusions were right: with adequate management and that eye-watering sum of $11 billion, it could have been successful.
However, as clearly illustrated in the diagram above, with so many moving parts, the HealtheVet project was highly ambitious. Trying to replace the Mumps-based VistA application with a Java/Oracle alternative in one fell swoop made it even more ambitious and risky.
Even if it had succeeded, there’s another consequence of all those moving parts and complexity: maintenance and ongoing development of the HealtheVet platform would have been extremely difficult, time-consuming and costly. Like any IT platform, it would be fine whilst things worked successfully, but one thing you can guarantee about IT systems is that at some point, a problem will occur. The more moving parts you have in the technical architecture, the more places there are for problems to occur and therefore the higher the probability that problems will occur.
So imagine a critical problem had arisen in some part of the HealtheVet architecture. How would it have been tracked down and who would have been capable of such troubleshooting? One look at that diagram above, and mere common sense should suggest that this would have been time-consuming and problematic at the very least.
Another thing you can guarantee about an IT application is that it is never cast in stone. At some point it will need to be amended and/or extended. Once again, the HealtheVet complexity would have made any modifications very difficult, time-consuming and expensive. Multiple teams would have probably have been involved, and the consequences of even small changes would have potentially rippled down through those many Java EE layers.
All this having been said, there’s nothing intrinsically wrong with the Java EE Platform, and as I’ve indicated earlier, there are many large organizations around the world that run successfully and depend on the Java EE Platform. In the IT industry it has been regarded as something of a benchmark architecture that offers demonstrable and proven scalability and reliability. Performance, whilst perhaps not exceptional as a result of all those intermediate layers, is usually more than adequate, particularly when the price/performance of hardware keeps increasing in line with Moore’s Law. As a result, it’s hardly surprising that Java EE is the IT architecture that tends to be recommended by the large management consultancies, and is the IT architecture for which most of the big contractors have lots of expertise.
In a nutshell, many, if not most, enterprises have come to accept the downsides of Java EE’s complexity against the benefits of its perceived scalability, reliability and ubiquity. A case of “nobody ever got fired for recommending Java EE”.
Over the last three or four years, an interesting change in thinking has begun to take place amongst some of the largest users of the Java EE Platform. Companies such as EBay, PayPal and Walmart, all of whom have relied on massive Java EE Platforms, have begun to replace them with a relatively new alternative technology called Node.js. This is all summed up nicely in this Infographic.
The benefits that these large organizations are realizing by moving to Node.js are nicely epitomized by PayPal’s experience. For example:
“The Node.js app was:
“.. the Node.js application had:
Similar findings are documented by the other large enterprises that have begun evaluating Node.js and have found that it has revealed shortcomings in their established Java EE architectures that are now beginning to make them question its position as the reference standard enterprise IT architecture.
If you want the difference in complexity between a Java EE architecture and a Node.js-based architecture to be thrown into stark relief, you need look no further than the two diagrams above: compare the EWD.js / Node.js / VistA architecture diagram with the HealtheVet diagram.
Well, let’s first sum up these various threads I’ve outlined above:
What’s wrong with the VistA Evolution picture is that it looks, to me, very much like a repeat of HealtheVet:
Why should such an approach to VistA Evolution succeed this time around when HealtheVet failed, no matter how much money and expertise was pumped into it? Surely that lesson should have been learnt, or is it the case that there has been a whole-scale replacement of senior management within the VA since HealtheVet was canned, and so there’s nobody left who even remembers HealtheVet?
What’s wrong with the VistA Evolution picture is that the knowledge and expertise of the Open Source VistA Community is apparently being ignored, despite the VA’s supposed commitment to the Open Source-based strategic direction that was instituted by Roger Baker as a direct result of the HealtheVet failure, and despite having set up OSEHRA whose very remit includes the creation of a two-way flow of ideas, solutions and best practices between the VA and the Open Source Community. Instead, it appears that solutions are being prescribed and contracts awarded to companies whose standard solution will be inevitably based around Java and the Java EE Platform, because that’s the skills they have available, not because it’s the most appropriate technology for the VA.
What’s wrong with the VistA Evolution picture is that those large contractors who are being awarded huge and expensive contracts are not being mandated to consult with the Open Source Community, and made to ensure that they don’t drive the VA into a repeat of the HealtheVet attempt to modernize VistA.
Industry leaders are moving away from Java and adopting Node.js.
EWD.js allows the VA to follow suit and modernize VistA with fewer moving parts and much less complexity, and therefore in a much lower-risk way. EWD.js will allow to VA to realize the same benefits that those industry leaders are reporting.
EWD.js allows the thousands of man-years of effort and skill that has gone into creating VistA to be handed on to the next generation of developers. With EWD.js, VistA, the EHR that consistently comes out top in surveys of user satisfaction, and that is universally loved by its users at the sharp-end within the VA, doesn’t need to be confined to the scrap heap. In the light of the evidence I’ve presented here, surely that would be a serious crime and an act of immense stupidity.
It’s time to fix the VistA Evolution picture. It deserves to be a masterpiece that can stand the test of time and ever-changing fashion.
Isn’t that what Veterans deserve?