Documenting Integration Agreements

Yesterday and today I'm focusing on the Fileman 22.2 Technical Manual. Sam has already done most of the work to update this manual with complete lists of all the new software elements George Timson and the rest of the team created, and Kathy Ice has already overhauled the formatting to put it into our document templates and styles. I'm now working on finding last-minute missing details - and in some cases major ones, things the VA documentation standards never required Technical Manuals to include, but that ought to be there anyway - and also things they did require but that are nevertheless conspicuously absent.

The first missing element is an explicit discussion of namespaces and numberspaces. When discussing a VISTA package's technical architecture, this is the single most important thing to know - where are its software elements? - but the current standards don't require it to be listed in a Technical Manual, and until now it wasn't. It's there now for FM22.2, right in the Introduction.

The second missing element is an overview of this package's dependencies on other packages. In theory, the Database Integration Agreements (DBIAs) list every point of contact other than supported references, but although necessary it's also too much detail for someone new to Fileman's internals. We need to add an intro that explains what other packages the current package relies on, and the general shape of those dependencies. In Fileman 22.2's case, it will rely upon 1995 Standard MUMPS and nothing else. At the moment it also relies upon two uncertified, unverified routines written by Joel Ivey that are not part of VISTA's primary codebase, but I'm fixing that today, so it doesn't need to be documented. However, for almost two decades File Manager has depended upon Kernel, and it was one of our goals for 22.2 to end that dependency. Fileman can now be installed without Kernel again, like it used to be, without causing any problems. That change is historic enough (and important enough to VISTA's overall architectural software stack) that we're documenting it in the new dependencies subsection of the Introduction, along with the 1995 Standard MUMPS requirement.

The third missing element is a requirement according to the current VA documentation standards, but one VA has finessed. Documentors are supposed to list all the integration agreements in the Technical Manual, so readers can study the specific dependencies between this package and other packages, but for quite some time the documentors have instead given instructions for how to look up the integration agreements in VA's Forum system - which most VISTA adopters outside VA have no access to. That's reasonable up to a point for readers within VA - the VA documentors' main audience - but only up to a point.

There are a couple unfortunate results of not explicitly including DBIAs in the Technical Manual.

First, it means no one outside VA really knows what those DBIAs are, or thinks about them much, or designs their own software lifecycles or architecture to take them into account - thus defeating the core reason Cameron Schlehuber set up the DBIA system in the first place.

Second, it means no one inside VA reviews the existing DBIAs on a frequent basis, which is supposed to be done so they can help drive package development toward a reduction of private DBIAs over time. Privately agreed upon undocumented accesses are supposed to be replaced with standard APIs, and the system overall is supposed to help us identify and eliminate overall system complications over time. Instead, it has become a check-box item to point to the DBIAs and occasionally make new ones, but otherwise to ignore them. I know this because many of the DBIAs for which File Manager is the custodian explicitly say they were supposed to expire long ago, but they're still active, though ignored.

This needs to change. I hope VA will seriously consider changing its documentation procedures to follow the documentation standard explicitly by listing all DBIAs for each package in its Technical Manual. To lead by example, that's what I'm finishing up for Fileman 22.2 today, and it's proving educational.

Going forward, these DBIAs will become one of the to-do lists that guides future development of Fileman, in this case toward simplifying its interactions with other packages and theirs with it, to create a more elegant VISTA architecture over time.