MIGRATION: OpenDocument is in My Future

Perhaps a turning point in my year, 2005, came in one instant in June at the Massachusetts State House at the ODF Summit hosted by Peter Quinn. It was at the point in proceedings when Quinn cast the question around the room, "What's our conclusion, then?" A few people spoke and then I heard IBM's Bob Sutor in his tan summer suit say, "OpenDocument is definitely in your future."

IBM's software strategist, Doug Heintzman (with whom I had sat on a panel that Winter where I heard him express views to the effect that OpenOffice's file conversion performance was not complete) who was sitting there next to Bob, was nodding in the affirmative. At this time (June 2005), I had not yet heard IBM's confident declaration of support for OpenDocument (something I had been supporting since 2001) and it was a startling moment.

Startling because I always knew that OpenDocument represented a change in the way many people do things. And in the cauldron of influence, if Big Blue says it's okay, then it really is OKAY! June for me represented that first moment I could see clearly that OpenDocument -- as a universal phenomenon -- was going to happen.

CIOs Will Agree

If you are a CIO or IT staff at the state or municipal government level (in any country), you too will be thinking, "OpenDocument, it's in my future." If you haven't had this thought yourself yet, you certainly will in coming weeks and months as you become aware of the number of your colleagues in different locations who are proactively dealing with this technical & cultural shift.

The news last week came in from the National Archives of Australia, from the US State of Minnesota, and from the CIO of the national government of Norway. They are all going down the OpenDocument path. The news can now only point to the inevitability of arguments in favor of a single, open file format standard for office documents. Each new adoption case weakens the social contract supporting the old proprietary way of doing things with documents. In time, any discussion at the state and municipal government level about the benefits of Microsoft's new format will prompt cock-eyed glances, the same kind we got when OpenOffice first launched its reference implementation of OpenDocument in 2001.

What is so reassuring about Minnesota is that they are quite aware of Massachusetts (of the policy, the policy process itself as well as of Microsoft's tactics there). And Minnesota's tactics show how much they have taken on board. These are not ignorant people. The wording of the legislation not only establishes a positive course toward OpenDocument for the state; but by making a more specific definition of "open" in the context of the document standard, Minnesota at once paints Microsoft's ECMA strategy into its destined corner while making it practically impossible for Minnesota agencies or Massachusetts policy to revert back to support for the second Microsoft standard when the ECMA and ISO processes conclude.

Minnesota's Influential Definition

Minnesota's rigourous new definition of "open standard" (context is the file format)...

"Open standards" means specifications for the encoding and transfer of computer data that:

(1) is free for all to implement and use in perpetuity, with no royalty or fee;

(2) has no restrictions on the use of data stored in the format;

(3) has no restrictions on the creation of software that stores, transmits, receives, or accesses data codified in such way;

(4) has a specification available for all to read, in a human-readable format, written in commonly accepted technical language;

(5) is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format;

(6) if it allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;

(7) allows any file written in that format to be identified as adhering or not adhering to the format;

(8) if it includes any use of encryption, provides that the encryption algorithm is usable on a royalty-free, nondiscriminatory manner in perpetuity, and is documented so that anyone in possession of the appropriate encryption key or keys is able to write software to unencrypt the data.

"Restricted format" means any data format that is accessed, stored, or transferred and is not open standards compliant.

Let me be perfectly clear about this: there is unanimous agreement among IT & standards experts as among state government IT officials that Microsoft's new file format does not meet the widely agreed definition of 'open' in the context of a format. MSECMAXML currently fails on at least points 5, 6 & 8 of the Minnesota Bill (there will be much more written on these points). After Minnesota, there will be no turning back away from a single, real, open file format standard: no turning away from a total, exclusive commitment to OpenDocument. The plain result of Minnesota and other states whose approaches succeed to further box Microsoft into its own bad strategy is that no state or municipal government can possibly argue or justify a non-OpenDocument direction. There will be no adoptions of the MSECMAXML among this class of organizations because the single-vendor dependencies in that format negate its contention. Do you recognize that MSECMAXML cannot qualify for consideration next to OpenDocument? CIOs: If the answer is 'yes,' please enter a comment. If 'no,' please enter a comment that helps us understand your issues & point of view.

As you recognize that change is inevitable across your organization, the same sequence of thoughts will come...

• How do we take the OpenDocument direction on?
• Shall we pursue the policy route (like Massachusetts), or the legislative route (Peru, Venezuela, Brazil, Minnesota)?
• What is the most efficacious way to influence all our agencies to migrate the format?
• How can we implement OpenDocument?
• What are my choices: OpenOffice.org 2.0, StarOffice 8, IBM's Lotus Workplace, Writely?
• Over time, what new options are likely to present themselves?
• What will happen with the users?
• Will they resist, even rebel?
• What about my legacy documents?
• What about macros, databases, mailing list processes & stubborn workflows (dependencies on enterprise applications) which need to be re-engineered?
• How shall I make the arguments that this action is necessary?
• Who must I convince?

If this sequence flashing before your eyes causes anxiety, you can be calm at the notion that others, too, are dealing with the same questions. Thousands of others. This means that the questions will be answered. Not only that, but the answers will be shared and improved through exposure and repetition. This is how the half-life of an ODF conversation near you -- from first meetings through implementation -- will diminish with each successive adoption case.

If we watch what happens at the Bristol City Council, at the Massachusetts Governor's IT department, in Norway and, if the legislation ends well, in Minnesota, then we will have templates, step-by-step instructions for action elsewhere. Each subsequent migration will be faster and more efficient than the last because we learn from successes as well as from mistakes. In time, again, we will come to recognize that OpenDocument migration is not so difficult, not so disruptive; that it is not so strange.

OpenDocument migration is what we did.

Comments

[Jeff Kaplan]

On the list of issues to pay attention to are:

Is accessibility for disabled adequate? (an issue for the moment).

Have I made the business case to political leaders and other stakeholders?

How do I ensure that open standards become a requirement for ICT procurements by public agencies?

Also, I note that the MN definition of "open standards" wisely omits one element often seen -- "reasonable and non-discriminatory terms" which is a veritable slippery slope to new kinds of lock-in. It also unwisely omits consideration of usage in the marketplace (provided all other openness elements are still met). Andy Updegrove blogged about this today, and rightly so.

[orcmid]

I can't figure out any way that the Open Office XML going through ECMA TC45 will come out with any more deficiencies as to Minnesota points (5), (6), and (8) than will the current Open Document Format. In fact, the ECMA draft definition of strict compliance seems far stronger that the ODF equivalent.

What, specifically, do you see that has the ECMA TC45 Office Open XML Document Interchange Specification necessarily fail at (5), (6), and (8)?

[Sam]

orcmid-

May I encourage you to read my post, "WMF & the ECMA Spec," for detail on the proprietary dependencies being implemented in Microsoft's format?

These defeat the definition of "open" as now given by Minnesota.

As for the points...

Point 5)
"...complete semantics..."
This language would be a problem for MSECMAXML if it the format includes binary or encrypted information -- such as DRM -- which fails the other requirements here.

Point 6)
"...extensions are documented..."
MSECMAXML -- the "ECMA Spec" -- reveals system calls to Office & Windows...much of which is undocumented. There may be other relevant examples I'm not mentioning.

Point 8)
DRM again...it's a no-no as presently specified.

[orcmid]

I checked your WMF & the ECMA Spec post. WMF is a line-drawing format of course, so it is not equivalent to JPEG (which is patent encumbered, but you knew that, right?). My understanding that the preferred form for images will be PNG or maybe TIFF and I'm not sure why some non-WMF vector graphics form is not being used for OOX as a new default within the strict compliance case. I haven't been following the SVG story.

But, clearly, they need a way to make completely legal, fully-specified OOX documents where only openly-specified/-standardized formats are required. I figure they'll fix it. (Let's not talk about the patent encumbrance on GIF either, OK.)

I don't think the issue of storing images in binary is a concern, so long as the format is public and unencumbered (e.g., TIFF without proprietary compression schemes).

PS: In the text search I just did of the TC45 base document, WMF only shows up in examples and e.g.s. Now, how people profile requirements to deal with legacy uses is not implied but some work may be called for to make OOX ready for fully-open usages. I checked the document for "metafile" also and the two occurrences that came up are both benign and could have been replaced by a different term(related to legacy use of Ole and advice on rendering thumbnails).

PPS: This is certainly a great area to pay attention to, especially in perfecting a way to ensure that one can limit/convert files to use only open+public formats in use cases where that matters for interchange and for preservation.

PPPS: I think the connection of an exploit against the WMF imager implementation (now corrected) and the use of WMF (or any image format) in document interchange is a reach too far. We've had malformed-JPEG exploits too.

[orcmid]

We're talking about a document format, right? Where are there reference to APIs in non-illustrative materials in the ECMA TC45 base document? (I imagine the Apple members will help root those puppies out, but what is the context for any that are there?)

[orcmid]

I'm baffled by point 5. There are complete semantics for TIFF, which is a pretty purely binary format, with compression. Is there a confusion about what semantics means in the context of computer formats?

Also, a format can be completely defined and have digital signatures and encryption (as in the XML-DSIG approach). ODF packages can be encrypted too. This doesn't seem to be contrary to open formats with open specifications. It might be contrary to a particular use case, and that should be regulated in the situation of use, seems to me.

I have no idea how DRM creeps into OOX, but I see where digital signatures are provided and encryption is designed for restricted materials. Is encryption taken to mean DRM or are you seeing something explicitly specific to DRM (as with media streams)?

This raises an interesting case. Do you insist that such uses of an open-format be impossible or simply that they not be required and can be made impermissible in particular usage situations?

I think this last is an important area to explore with regard to all broad-use formats for document interchange.

[Done for now. The IRS has some document formats I must go honor now.]

[Sam]

orc-

You have some interesting questions that deserve sincere answers. I know there will be many people interested.

This is a physically poor forum for it, so I will try to address them in a useful format.

[Sam]

I'll unfairly address your questions with a few questions...

Among the general questions I would urge readers to consider is, for example, what would a Linux-based office suite implementation of MSECMAXML do with system calls like WMF that are intended to invoke processes in Windows? Implemented on a non-Windows platform these are dead processes at best, vulnerabilities at worst; but a truly open format would not have calls that work for one vendor.

Also, what of DRM? This is a binary blob sitting in the midst of a file that non-Windows platforms cannot swallow; it defeats the purpose of an open standard format.

One response from a Microsoft developer I remember seeing once was that an OpenOffice could maybe just delete the binary and carry on. This, however, would impact the Microsoft license...and there goes some data. This is a non-starter.

[orcmid]

I agree about finding a better form for addressing some of these questions.

I don't understand your reference to Windows Metafile (WMF) as if it is an API or system call. It is a data format used to carry graphics and that was the only use I saw in the examples that I found in the ECMA TC45 draft. I think there is some kind of mechanism that allowed software-behaviors to be invoked (like macros elsewhere perhaps, I'm being lazy here and not going to look it up), so obviously one would want to ignore those in an interchange situation, and preferably prohibit them. I guess we should find out exactly what that is about.

The presence of DRM has nothing to do with being a binary blob though it certainly has to do with not being able to access the encoded content without a suitable license or other claim check. This is different than encryption for privacy and limitation of access purposes.

And either way, are you saying that should be impossible or that it is inappropriate in public documents used in eGovernment? (I assume you won't mind if the CIA and NSA use OOX, though?)

We need to sharpen our terms here. But I think the bigger question is whether or not the format should prohibit certain uses or there would be prohibitions by the parties to interchange under particular conditions, and having it be easy to establish and confirm satisfaction of such restrictions.

I think this matters, by the way, for all candidates for open standards to be applied under the conditions specified in the proposed Minnesota Information Architecture requirements and it is very valuable to consider this question independent of what those candidates are.

[Sam]

The question, orc, is what is appropriate behavior of an open standard file format.

There happens to be a living, working, reference implementation of one already: OpenDocument in OpenOffice.

No one will ever know what MSECMAXML will do until there is a reference implementation, with open code that people can look at -- in its entireity.

This is why the Microsoft choice to launch its own standard is odd. They would not do so in order just to re-create another ODF. Their principal motivation to preserve special behavior which favors their app & OS platforms.

The ECMA specification will tell.

Two things are necessary: 1) interpretation of the points in the Minnesota Bill; and 2) an ongoing re-examination of what Microsoft is delivering in the ECMA Specification.

[orcmid]

I will concede that OO.o is a demonstration of an implementation, and that's important. That it qualifies as a reference implementation is not clear to me. (They use non-standard extensions, for example. They actually have to at this point.)

I certainly agree that there should be on-going scrutiny of the ECMA TC45 drafts and we should sharpen what the open-ness test is in different settings and how that is handled in practical realities with multiple offerings of an interchange format.

[orcmid]

I just ran into this blog which speaks about some uses of encryption and digital signatures that are rather critical in business and government contexts: http://blogs.msdn.com/recman/

The minute the private key used in a digital signature is compromised, every use of that key is suspect and repudiatable.

Of course the public key gets to be known, and the algorithm for the encryption is also known, but the private key must remain a secret and digital signatures must be possible in electronic documents in order to preserve their authenticity and prevent manipulation.

This should all be permissible in Minnesota. I can't imagine how it couldn't be.

Do you find this unacceptable? I'm trying to refine the calibration around "DRM" and also binary but understood content.