FLAP and Fileman's Architectural Role

To say that File Manager is VISTA's database management system is a gross understatement, so gross that it is more misleading than helpful.

Traditional database systems run side by side with the applications, offering data-related services (like creating, reading, updating, and deleting records - the services with the unattractive acronym CRUD - along with many, many other such services). In other words, most DBMSes have a peer-to-peer relationship with the applications that use them to manage their data.

Most architects, managers, and programmers who lack deep VISTA expertise would describe Fileman's relationship with other VISTA applications the same way, in part because they think of Fileman as a DBMS, and because that's what most DBMSes are like. Because Fileman's relationship with the rest of VISTA is so very different from that, its best not to think of it that way, even though DBMS services are an essential part of what Fileman was created to provide. VISTA hardhats with deep experience will often speak about Fileman the same way, because we lack the terminology to say what it really is, but we know better, and we see Fileman more like it really is - the beating heart at the foundation of VISTA's architecture. Just because we call Fileman VISTA's DBMS, that doesn't mean that's all that it is, or even that it's the most important part of what it is. The truth is far more interesting, even if we rarely put it into words.

Yes, Fileman describes and manages almost all of VISTA's data; the only exceptions to this rule require explicit exemptions from VA's Standards and Conventions Committee to be allowed to bypass Fileman. That sounds a lot like a DBMS. But Fileman also does two things most DBMSes don't do, and they make all the difference.

First, some of the data Fileman stores is software. Although that makes up a small part of what Fileman stores, it's a very important part. Most VISTA software is ultimately called from these software data fields stored in Fileman. Options, remote procedures, web services, protocols, tasks, and other kinds of VISTA programs are all invoked originally from these software fields in Fileman. Fileman, therefore, acts as the master library of VISTA software. Every different kind of VISTA program is defined by Fileman, and its entry points are recorded in Fileman, which also plays a role in dispatching almost every body of software that executes within VISTA.

Before I describe this further, take a moment to let that soak in. No database theory textbook I know of would include these kinds of architecture-definition and software-dispatching services within the traditional roles of a DBMS. This is a different kind of thing altogether; I invite you to think about what you know about computer science and propose some names for the kinds of systems that more traditionally have this role. In VISTA, it's Fileman that has it, along with extensions to Fileman that have been bundled up into Kernel and other VISTA infrastructure applications.

So Fileman cannot be "just" a DBMS in the way most people mean when they refer to it as such. And make no mistake, the word "just" or "only" is always implied strongly and strategically when people dismiss Fileman as a DBMS. Framing VISTA's architecture and helping with software dispatch is not "just" some additional oddball add-ons to Fileman, of minimal interest. No, this is what VISTA is made of.

Specifically, any VISTA application - like Lab - is not an application or package or program in the way almost everyone means those words when they use them. You cannot install just Lab on a computer and get an operational Lab system. It just lays there, inert, unable to function. And the problem is not that it's missing some friends (like a DBMS) to talk to, to offer it a few missing services, to complete it. It can't be completed. It's not a module, or a package, or an application, or any other kind of thing that we are used to from elsewhere in the software world. VISTA Lab is "just" a collection of thousands of micro-modules designed to plug into the Fileman framework, to teach Fileman how to support a Lab department. Lab is a recipe book, a suite of instructions. VISTA Lab does not perform Lab operations. Fileman and Kernel do. Lab just teaches them how.

This is not just a very, very eccentric way of thinking about VISTA applications and architecture. It's also a far more accurate way, and it lets us understand some deep truths about what is or is not possible with VISTA modernization that are otherwise invisible and even unthinkable to us. If Lab is not a separate or separable program, then plans to "modernize" Lab by excising it and replacing it can be seen as 100% missing the target. Excising Lab from VISTA would be like excising your lymphatic system from your body: good luck; you'll need it.

When we think about VISTA applications as Legos to be snapped in and out, we think we're playing Chess when we're actually playing Go. The board is different, the rules are different, the pieces are different, and we're guaranteed to lose. Likewise, when we think about Fileman as (just) VISTA's (replaceable) DBMS, again we're missing what's actually happening, and all our misguided policies and plans will collapse when they run into the very different reality of how VISTA is organized, and the central role Fileman plays in that.

Second, Fileman has the ability to knit together code and data in limited but surprisingly powerful ways. Using "cross-references" - a name that reflects the original intent of this capability, which has long since grown far beyond its original intentions - file designers can attach behavior to database updates, allowing the data-driven invocation of behavior. When a record is created, selected, updated, deleted, or more, additional behavior can be defined by the file designer to take place. More impressively, without breaking or colliding with the file owner's intentions, any other programmer can extend that file to add more behavior, their own behavior, of interest to them. Fileman thus offers an arbitrarily extensible system for linking any behavior to its "database" operations, allowing for the creation of a genuinely data-driven architecture.

This is the opposite of the legacy relational model, in which data consists of notes made in tables, existing passively, to be operated upon by software that knows why the data exists and what to do with it. In such models, data is dead, helpless, mere matter to operated upon by the brains of the system. Most plans for modernizing or evolving VISTA that are proposed presuppose this dead relational model of VISTA's data, but VISTA is not like this.

VISTA is more like the object-oriented model, in which data and code are organized together in ways that model the real world. Instead of putting all the dead data in a database and all the live code that operates upon it somewhere else, the OO approach is to put code and data together, grouped by what they describe. So all the data and all the code that describe patients are stored together, and all the code and data that describe lab tests are stored somewhere else. The wall between code and data is broken down and no longer treated as the first organizing principle. Instead, the real-world entities being modeled by the system are treated as the first organizing principle.

Since the data and the code describing, say, a lab test are more closely related in an OO model, they can interact more closely. The data can know about and even invoke the code, which in turn can better protect the data. The data therefore becomes an active player in interacting with the rest of the system, unlike the relational model. If you have the authority to access relational data, it can't stop you from doing the wrong thing to it, because it's just passive notes. But if you try to do the wrong thing with OO data, the code that gives you access to that data can intervene to stop you - even if you have authorized access - because the data has code it can invoke whenever it needs to check on what you're asking it to do. This offers powerful additional security and database validation capabilities.

Put another way, in a relational model code is code and data is data. They are completely distinct and behave in stereotypical ways. But in an OO model, code is also data (defined in the object's "class") and data is also code (exposed as "properties" instead of as fields in a table). You can't get to OO data except through its protective membrane of code, and it in turn can invoke any of its code capabilities it sees fit to service your requests. The property you ask for may not even exist but may be calculated on the fly; to the outside world it looks like a simple field, but it's actually a blending of state and behavior that's capable of taking an active role in your interactions with it.

To a programmer raised in a parochial relational context, this autonomy of OO data is irritating - they want data to submit and obey, not talk back or make their own decisions - but that model of passive data is far too error prone to build highly complex systems with. Just like an employee who takes no initiative and does precisely what you say, you may at first imagine it's a good thing, but when you realize you're responsible for every single decision it makes, you quickly realize you've heaped all the hard work on your own back. Keeping track of the thousands of interactions and validations necessary to do the job correctly may emotionally gratify control freaks who resent other people's autonomy, but the vision of a perfect programmer or designer who rapidly flies through thousands of steps without error is a mirage; the more details we have to manually get right, the more errors we must and will make, and eventually as the complexity of our systems rises the number of errors becomes catastrophic.

This is why OO methodologies arose as an alternative to the relational model. The blessing and the curse of modern software is our ability to rapidly create systems too complex for us to manage. The complexity lets us precisely model highly complex real-world problems like medicine, but it also raises the difficulty of accurate and precise operations beyond the human capacity to comprehend, leading to errors. OO's reason for existence is to hide most of that complexity, to limit our access to the system, to organize the validation, security, and other operations in more intuitive ways, and to limit our access to the data to only the proper ways of doing things. We lose control-freak kinds of control, but we gain effectiveness kinds of control. As our data becomes smarter about itself, more self-aware, more involved in validating itself and automatically coordinating with other fields and other applications, our workload drops back down to something sane, something we can succeed at.

But this all requires letting the data have more autonomy, more control, than someone raised strictly in the relational model - which we often see in people's visions for service-oriented or three-tier architectures - is going to be comfortable with. Fileman can store data in relational ways, but that's like hiring Einstein to run a cash register. Fileman is not an OO database - as a student of OO for decades I'm skeptical about whether any database today that purports to be OO has even come close to realizing the huge potential of this idea for databases - but Fileman does use some OO principles in how it organizes data and code, including the ability to attach behavior to its data. VISTA's architecture depends on these close relationships between its code and data that Fileman makes possible, depends upon Fileman invoking the right behavior at the right times to ensure the whole system continuously achieves medical, legal, logical, and database integrity. Fileman helps knit the whole VISTA system together, out of the data, out of the core VISTA extensible engines, and out of the tens of thousands of micro-modules that the application developers have created over the decades to instruct those engines about how to support medical care.

Is Fileman a DBMS? Sure, in the same way a human being is a mammal. But just as you are not "just" a mammal - few horses have ever been elected to serve in a senate - so Fileman is not "just" a DBMS. The differences aren't just interesting; they're the whole game.

This is what the FLAP Jordan Team went over this morning, along with an introduction to Fileman's structure.