Open Standards Mandatory in Denmark

John Gøtze kindly brings to our attention a new development in the Danish government's progress toward open software standards.

Now, having gotten past the preliminaries of whats, wherefores, hows & whys, the Danes have progressed to the question of implementations. This always reflects a watershed moment for government leaders, staffs & citizens who will be effected. It reflects a body of people coming through a process of understanding and it shows their confidence.

This reminds me of nothing more than Peter Quinn's meeting of vendors in the Massachusetts State House in June of 2005 where he said,

"Open document formats: I get it! But how do I get there? Discuss."

The difference now is that we are dealing with a whole country, Denmark, which according to scale is something like the size of Massachetts within the larger body, the European Union. In contrast to the Massachusetts situation, other parts of the EU are already migrating ahead of whatever policies or regulatory guidelines are being established. These include agencies in France, the UK, Germany and Belgium among others.

There is an interesting similarity to Massachusetts. The policy memorandum, ETRM 3.5, which fostered the ODF debate there was similar to this Danish plan in its underlying motivations and intent.

When a government gets past identifying the ideal scenarios that are possible, those which exist and are ready to implement -- in this case they include open standards like XHTML, CSS, ODF -- and moves on to the questions of how to get them used, there is always a large number of impediments to the final result. These include recalcitrant software monopolies (who are constantly trying to undo good policy work), general inertia against change, decentralized structure of multiple agencies with different ICT systems, leadership and beliefs about what works, and the difficulty in establishing an authoritative, credible but also flexible recipe for pushing change without increasing cost, stress & disruption.

That's why these policy frameworks look so alike: every bureaucracy gets to this same difficult place eventually -- 'How do we get there?'.

Says John Gøtze...

The implementation plan is presented in a report which suggests that “open standards should be implemented gradually by making it mandatory for the public sector to use a number of open standards when this becomes technically feasible”.

What has happened?

The existing Danish Interoperability Framework (in Danish) has become mandatory. Separately, the report lists a number of open standards which should be implemented by Jan 1, 2008, through the normal course of system upgrading (unless the transition is deemed disruptive).  Gøtze mentions a few...

• Standards for data interchange between public authorities
• Standards for electronic file and document handling
• Standards for exchanging documents between public authorities (Open Document Format and Office OpenXML)
• Standards for electronic procurement in the public sector
• Standards for digital signatures
• Standards for public websites / homepages
• Standards for IT security (only within the public sector)

Around a dozen standards: Compliant XHTML or HTML, complaint CSS, WAI Level 2, OCES (digital signature), XML 1.0, XML Schema 1.0, NDR 3.0, FESD (docuument management), OIOUBL, UNSPSC, and DS484 (ISO 17799).

We're interested here in ODF. Here's what the report says about document formats...

With regard to standards for exchanging documents between public authorities, the report proposes that “it should be mandatory to use at least one of the document standards Open Document Format or Office OpenXML”, and that it is up to the individual agency to decide what they want. The report explains that a study will be conducted this year with “the purpose of obtaining the necessary experience with these standards before 1 January 2008″.

So, the Danes are looking at a mandatory shift to either or both of the two XML-based document formats. You say ODF AND MSOOXML BOTH! EEEEEEK!

This gives me no anxiety whatsoever. MSOOXML has already been thoroughly de-bunked vis a vis its repudiation of other existing standards; it is in perhaps a perpetual deep-freeze at ISO (from which Microsoft will not seek or wish to remove it, since "ISO status pending" is all they need to sell it; the alternative is to re-wire their entire new catalog of software); under further use testing and scrutiny in Denmark, its repudiation of the basic intentions of XML will be highlighted; and under scrutiny in Denmark the thorough dependency of MSOOXML documents upon the Microsoft stack (Vista, Exchange, Sharepoint, Outlook, MS SQL Server, IE7, Office 2007, Groove, etc.) and their lack of interoperability & compatibility outside the new Microsoft stack will be underscored and well understood. The Danes will find that MSOOXML is no solution.

Regarding document interoperability the Danes will learn on their own about the three possible solutions, two of which are free and one still requiring funding...

1) the Microsoft-Clever Age-Novell "MCAN" Translator (at sourceforge and being integrated into only Novell's version of OpenOffice.org)

2) the Sun Microsystems Plugin (still in development and promising document exchange fidelity equivalent to OpenOffice.org, which Massachusetts originally deemed inadequate for its decentralized migration)

3) the OpenDocument Foundation's Interoperability Suite (not funded but promising 100% document fidelity delivered through a) Plugin [Windows client], b) InfoSet API [server] and c) OpenOffice feature-set governor [OOo-side client])

Comments

[Gary Edwards]

Hi Sam,

Thanks for another fine commentary. For the life of me i don't see how mandating both ODF and OOXML will be helpful to any government. The quality of XSL transformation filters between the two XML file formats is so bad it might as well be zero. Although Microsoft assured Massachusetts that such a transformation would be trivial, nearly two years later they have nothing to show for their braggadocio.

Which means Denmark will be hoist on the petard of having their documents in two file formats that are not interchangeable. And except for the MSOffice ODF Plugins from Sun and the Foundation, application interoperability might as well be zero.

It's important to keep in mind that both daVinci and ACME 376, the Foundation's Plugins, were working fine in every beta release of MSOffice 2007, but broke in the final public version. I suspect Sun had the very same problem we did. It seems that Microsoft altered the system default for “zip” file format packages such that whenever a “zip” package like ODf is double clicked, the OOXML conversion engine automatically starts to convert the package to MSOffice 2007 in-memory-binary-representation. Of course, the OOXML conversion engine knows nothing about ODf. So the problem we now have is reverse engineering what looks to be a low level system “default” setting that loads the OOXML conversion engine instead of the daVinci conversion engine.

Microsoft has a long history of dirty tricks and an ever moving APi, effectively protecting their monopoly from would be poachers and thieves, er “competitive application and service providers working the Windows – Vista platform”. And here we go again.

This first act of aggressively blocking the ODf Plugins should signal governments loud and clear that moving into the Vista Stack of desktop, servers, devices and services is going to be the equivalent of exclusively mandating OOXML. We've known this to be the case in the past, the primary example being the Exchange/SharePoint Hub and developer platform which is optimized for MSOffice 2003 MSXML. So much so that the E/S Hub has to be considered an ODf killer.

Yet even those governments exclusively mandating ODf are boldly going forward with E/S Hub purchases, totally unaware of the consequences. As i've said many times, Massachusetts is just one E/S Hub Court Docket System away from revoking their ODf requirement and standardizing on OOXML. (Watch carefully now, the hand is quicker than the eye; ViSTO 2005, which was released with MSOffice 2007, dropped support for MSXML entirely in favor of the MS version of OOXML. (i mention this because there is clear evidence that MOOXML, legacy MOOXML, and now MOOXL Binary InfoSet for Excel all include eXtensions and dependencies that differ from the Ecma 376 version submitted to ISO/IEC).

I'm in agreement with Peter Quinn , "Open document formats: I get it! But how do I get there? Discuss.” (OBTW, great quote Sam)

The simple truth is that the ODf Community is not providing the means to get to ODf. There is no “bridge from the legacy installations of MSOffice and the billions of binary documents, to ODf ready applications and services.

Get to XML:

What most ODF-OOXML warriors forget is that the real issue for workgroup and workflow oriented consumers is getting their binary documents into XML. Which XML, ODf or OOXML, is a secondary consideration. The one thing everyone understands is that the way to connect important information domains, systems and architectures is through the skillful use of open XML and open Internet technologies. Important information domains and architectures includes:

Desktop Productivity Environments (the Office Suites)
Enterprise publication, content management and archive systems
SOA – Service Oriented Architectures
SaaS – Software as a Service systems
The Internet – Web 2.0 and beyond

You can't efficiently exchange information across these domains if it's trapped in application – platform specific and unstructured binary formats. Conversion fidelity requirements will break a binary bound exchange process every time. Even Microsoft realizes the need to move to XML, although they will of course strive to control that XML through continued use of application and platform specific dependencies and, artificially contrived implementation constraints.

XML Migration:

The problem as we saw in Massachusetts, Munich, and Bristol is that there are three stages of XML file format migration. The three stages for migrating to XML file formatsare inextricably linked and must be tackled in exactly this order:

1)Conversion Fidelity - the billions of binaries problem
2)Round Trip Fidelity - the MSOffice bound business processes, line of business integrated apps, and assistive technology type add-ons
3)Application Interop - the cross platform, inter application, cross information domain problem

Sadly, the ODf Community is near entirely focused on stage three, leaving the critically important first and second stages up to customers.

Interestingly, OOXML nails all three stages perfectly, with the proviso being that all interop applications be Microsoft application – platform specific. This is what they mean when they say, “interoperability by design”; applications designed to speak fluent OOXML, making exclusive use of .NET 3.0 system dependencies.

Note well that since the starting point of XML migration is that of MSOffice bound binaries and business processes, OOXML is today the only XML file format proposal that can perfectly answer all three stages of XML migration requirements. The OpenDocument Foundation does have three products in the works that are designed to also perfectly answer these same XML requirements, but the Foundations approach has met with considerable resistance, argument, doubt and continued outrage from the ODf Community at large.

Note also that these three stages are only important to workgroup and workflow situations – situations where MSOffice bound business processes are critical to day to day operations. People who are involved in simple document exchange can easily move to ODf through an OpenOffice download and some simple conversion artifact fixes as documents are converted as needed. Workgroups and workflows require a very high level of round trip fidelity. Which is why mixed application environments present workgroups with an near insurmountable problem; even where the mix is limited to that of different MSOffice versions!

Of course no one wants to see the life span of MSOffice eXtended indefinitely, which is exactly the conclusion most come too regarding the daVinci Plugin. daVinci makes it too too easy to stay with MSOffice since there is zero disruption to MSOffice bound business processes, line of business integrated apps, and the functioning of assistive technology add-ons.

The ODf community doesn't realize three things.
First, it's an absolute real world requirement that the three stages of XML migration be met perfectly by any file format contestant.
Second, Microsoft has handed the ODf Community the opportunity of a lifetime regarding the most difficult stage for ODf; stage two, the migration of MSOffice bound business processes.
Third, consumers migrating to XML have no stomach, tolerance or funds for the highly disruptive and costly rip out and replace” approach most in the ODf community favor.

The Massachusetts RFi:
Massachusetts ITD deserves credit for figuring this out. The three stages are the reason behind their ODf Plugin for MSOffice RFi. Which is about as desperate a cry for help from the ODf Community as i've ever seen. There is no other way to describe the events in Massachusetts except to say that the ODf Community refused to listen, and worse, made no attempt whatsoever to understand the exact nature of problem consumers wanting to migrate to ODf face. Hence the unprecedented RFi (Request for Information).

So what's going on here? It's the reality of being caught in the clutches of a monopolist for over ten years.

In 1995 Microsoft won the Office Suite application wars, with the next two years mopping up as they paved the way for the real crusher, MSOffice 97. The office suite wars were marked by the vicious marketing of comparative feature sets and competitive price inducements. The feature wars ended in 1995, were set by 1997, a time period during which a new movement had begun – the use of the ubiquitous MSOffice as a development platform for business processes, line of business integrated applications, and a proliferation of add-ons like those for assistive technologies.

In short, by 1997 it was no longer about “features”. It was all about MSOffice bound business processes. And the binding occurs at two levels; the MSOffice application layer, and the binary file format layer. This creates a binding trifecta of MSOffice, MS Binary Documents, MSOffice bound business processes.

The real battle today is over how to migrate or participate in the MSOffice bound business processes developed since 1995.

This is the core of the problem every organization trying to migrate to ODf must confront. Massachusetts, to their everlasting credit, came up with a rather magical solution; the idea of using ODf Plugins for MSOffice to solve the problems of stages one and two.

The stage three ODf problem of application interoperability is a work in progress, and i would direct everyone's attention to the ODf 1.2 progress of the three OASIS ODf sub committees; Accessibility, Open Formula, and Metadata RDF/XML. A great leap in application interop is brewing there, and expectations for ODf 1.2 run high with good reason.

The Problem With Rip Out and Replace:
Many people mistakenly believe that migrating to ODf is as easy as downloading for free OpenOffice 2.0. Which is true if your collaborative work is that of simple document exchange. For workgroups and workflows currently based on MSOffice bound business processes and binaries though, the disruptive and re engineering cost of moving those processes to ODf ready OpenOffice is impossibly high. While it's true that Novell is working on an automated means of converting bound business process VBA scripts to an OpenOffice footing, it's not there yet. But that's just a fraction of the problem. These business processes include compound documents for templates, forms, formula bound spreadsheets, and reports - often involving systems dependent data bindings.

Rather than trying to unwind these MSOffice bound business processes and re write them to somehow become application independent and ODf ready, we believe there is another approach demanding consideration. Instead of migrating to an ODf ready desktop MSOffice alternative, we believe the ODf Community should instead be focused on the Microsoft migration of these same business processes to the OOXML enhanced Exchange/SharePoint/Groove Hub.

The Great Opportunity: The Vista Information Processing Chain
Briefly the greatopportunity is this: Microsoft has been very busy migrating those MSOffice bound business processes to the Exchange/SharePoint Hub, which will soon be joined with Groove Collaboration servers.

XML Hubs are interesting in that they are an indispensable core to any SOA effort. The Hubs are a very effective point where access, aggregation and re purposing of XML services and information streams connecting to such things as backend data, transaction, inventory and billing processing or content – archival management systems. As a universal transformation layer able to connect to many disparate and purpose specific backend legacy systems, XML is unparalleled.

But what if these same XML Hubs are integrated as a way station connecting desktops and devices to Internet server systems as well as the legacy backends? And there my friends is the magic of an Exchange/SharePoint/Groove Hub. On the desktop side E/S Hubs integrate directly with the MSOffice productivity environment using OOXML (versions Ecma 376, MOOXML, legacy MOOXML, and MOOXML Binary InfoSet) as the both the XML container and transport. The E/S Hub provides the integration point where eMail, documents, workflows, project participants (people), and information resources (web services, XHRequest streams, media streams, and message streams) can be aggregated, sorted, re purposed, published, managed and scheduled. At the E/S Hub it's easy to bind data from MS SQL Server systems, MS CRM, MS ERP and MS Live to the portable OOXML document transport.

The most important point about XML Hubs is that migrating business processes to them always results in extraordinary productivity gains. Yes, the E/S Hub represents a new lock-in point for MSOffice bound workgroup customers. The immediate productivity gains however will far outweigh the long term cost of having your business lock-in to MS only applications and services for the next fifteen years.

So where's the opportunity you ask?

By intercepting this migration at the head point, MSOffice, converting the documents to ODf, and provisioning ODf ready Hubs the ODf Plugins could potentially walk off with the entire monopoly base of over 500 million MSOffice bound desktops. The ODf Plugin route is infinitely cheaper in that there is no need for desktop upgrades. And, the cost of long term MS lock-in is completely negated.

The ODf Plugin alone however isn't enough - which is exactly why the Foundation began development of the lightweight, efficient and highly portable ODf InfoSet Engine and APi. We need ODf ready Hubs to complete the ODf Information Processing Chain alternative. ODf Hubs that can compete with E/S Hub Juggernaut.

Cmon Alfresco, Lotus, and Zimbra - the whole world is waiting.

One more thing about these ODf Hubs and ODf ready applications that span desktop, servers and device platforms. The Internet has ushered in a new world of universal connectivity, exchange and collaborative computing. Some ODf ready applications will be designed to participate in specific information processing chains, so they will be built to act as efficient routers of information.

Other applications will profess to be ODf ready, but continue to act in the legacy traditions of information end points - ignoring the ODf document needs as defined by other information processing chains; dropping objects and data binding mechanisms they don't understand or lack the feature sets needed to make use of. Or, they might add value by asserting application or platform specific dependencies that would otherwise corrupt the documents use in other information processing chains.

Application Interoperability
The problems of application interoperability are far more difficult for ODf than for OOXML because ODf was designed for a very different objective.

OOXML was designed exactly for MSOffice bound binary and business process compatibility, (stage I & II of the XML migration requirement) including application and platform specific dependencies.

ODf was designed to be an application and platform independent universal file format, dependent on open XML and other Internet technologies available to any application or service.

For ODf, proprietary or platform specific dependencies are an interop killer. For OOXML, proprietary and platform specific dependencies are the monopolists life blood of interoperability by design”.

The world is going to move to XML, and from there to RDF/XML. That's a given. OOXML has the incredible advantage of meeting the three ”migration to XML” requirements. But the migration comes at the cost of long term business process lock-in, with interoperability demands limiting applications and services to the Vista Stack of MSOffice <> OOXML <> IE <> ViSTO .NET <> E/S/G Hub<> MS Active Directory <> MS SQL Server <> MS CRM <> MS ERP <> MS Live etc. etc. etc.

ODf can similarly meet the the three ”migration to XML” requirements, and do so just as well as OOXML – in spite of MS claims otherwise. Although they anger and upset the rip out and replace bent ODf community crowd, the ODf plugins for MSOffice from Sun and the Foundation are the only way ODf can crack and perfect these critically important ”migration to XML” requirements. Danes please take note!

And of course, there is today no ODf Stack or ODf information Processing Chain comparable to the one Microsoft has unleashed. Some great stuff cooking with Lotus Notes, but they are obviously lacking a strategy for stage I & II of the ”migration to XML” requirements. Alfresco is hard charging, but they have that same problem. Zimbra and Google Hubs are still mired in the application as end points approach where they see conversion of documents as a one way process where continuing and persistent loss of fidelity and the dropping of feature related objects or bindings is the expected and unavoidable collateral damage cost, so get over it.

Meanwhile, the ISO/IEC consideration of Ecma 376 is a diversion from where the real action is; the optimized for MSOffice OOXML Exchange/SharePoint juggernaut.

It's high noon for XML. Do you know where your ODf information processing chain is?

~ge~