About Distributing MSC Fileman 22.2 Using KIDS

Here's a little background to understand yesterday's reference to the chicken-and-egg problem of trying to create a KIDS distribution of MSC Fileman 22.2.

DIFROM and Transfer-Merge are Fileman modules, two of several dozen engines that make up Fileman's architecture. All Fileman's core engines - including DIFROM and Transfer-Merge - are intensely abstract and extensible, driven by plug-in software extensions.

George Timson created DIFROM (pronounced DYE-from) decades ago to be the heart of VISTA's software distribution and installation system. DIFROM is software a programmer runs on a VISTA source system that contains the VISTA data, metadata, and non-routine software to distribute. It asks what to distribute, then writes self-extracting MUMPS routines (called INIT routines [pronounced i-NIT]) to send to VISTA destination systems. When a destination system runs INITs, they recreate the source system's database elements.

These days, we rarely use DIFROM to distribute VISTA data; we use KIDS, a newer system; or rather, we do not use DIFROM directly, but we still use it all the time, as discussed below.

George Timson also created Transfer-Merge, which supports copying or moving files and records and merging files within one VISTA system at a time. We still use Transfer-Merge to assist with management of Fileman databases, but its most common use (unbeknownst to most users, programmers, or system managers) is to support KIDS.

Kernel 8 introduced KIDS in July 1995. KIDS depends on Fileman 21 work (released February 1995), especially an extension to DIFROM called the DIFROM Server, which broke up DIFROM into modular, callable components. KIDS calls those components - and calls into Transfer-Merge - to support both the distribution and the installation sides of its work. Any time the database is updated by KIDS, it's actually Fileman doing it at KIDS's request.

The DIFROM Server APIs are not for general consumption, because it's so very easy to screw up the great dance of VISTA software distribution and installation. Only Fileman and KIDS can call the DIFROM Server, but the engines themselves are extensible, driven completely by the software, data, and metadata they are distributing and installing, and executing plugin extensions as needed.

KIDS is extensible, but not as extensible as it could be. KIDS is table-driven, but the kinds of software elements it can distribute and its methods for managing each kind are in a hard-coded table. We want to modify KIDS to move this table out of the routine lines where it currently resides and into a Fileman file, to make it more completely extensible, instead of its current half-and-half state. This is an easy change to make - for those who know how - because the code otherwise is already primed to be extensible in this way.

When we try to distribute MSC Fileman 22.2 using KIDS, then KIDS will be calling the DIFROM Server and Transfer-Merge to bundle up DIFROM Server and Transfer-Merge (and other Fileman engines) from the source system, and then calling DIFROM Server and Transfer-Merge at the destination system to install DIFROM Server and Transfer-Merge (and other Fileman engines) into the destination system. Whew!

It's the latter part where it gets tricky. If we change the code and data we're using to install the code and data - and not changing them magically all at once, but gradually, in a series of steps - that could be vulnerable to mismatches between the old running code and the new incoming data and code, especially partway through the update, when the system has a foot in both worlds. That's where we're concerned about trouble, but we'll see what happens.

Although most VISTA software today is distributed by KIDS, we do still maintain special INIT routines for installing Fileman itself. Most of these INIT routines are not generated by DIFROM (though pieces of them were over the years), but instead are built by hand, to avoid the very chicken-and-egg problem described above. When you're bootstrapping the bootstrapping system for VISTA, automation is no replacement for expert hand-crafted solutions.