Introduction and motivation

This page describes a proposed architecture for WikiEngines.

The contents of a wiki is just a bunch of text documents, with version histories. Why can't you use software for dealing with text documents with wikis? Why can't you edit a wiki page in your favorite word processor? Why can't you grep through a wiki with the program "grep"? Why can't you move a wiki page from one wiki to another by dragging it to a new file folder, or with the command "mv"? Here's a way to do that.

The key is that, in the innermost workings of today's WikiEngines, the wiki pages are already treated as text documents, for the most part. It's just that other programs aren't allowed to access them in this way. Other programs are only allowed to access the wiki pages through the web interface. All we need to do is to give other programs the same degree of access that the WikiEngine already has.

The basic idea is to break a wiki down into a cluster of simple, interacting web services ("modules"). Now, wikis are already pretty simple, and there will be complexity introduced by networking these parts together. But by exposing more wiki internals as open, interoperable APIs, wikis will become more extensible. Instead of having to get the host to upgrade hir software to add a new feature, many new features will be able to be added client-side. Features like mounting the wiki as a file folder. Or a WikiRefactoringBrowser.

Everyone will still view the same pages when they visit a given wiki. But different users will interact with these pages in different ways, by running client-side software.

A less radical thing that you might do with this freedom is choose to browse an OddMuse wiki through a MoinMoin web interface.

Decouple the frontend from the backend

The database (the backend)

At the heart of every wiki engine lies a database which stores the pages, and the page histories. In many wiki engines, the database is actually a program within a program. The rest of the wiki code asks the "database part" to store and retrieve pages and page histories.

The functionality of the "database part" of many wikis is in fact quite similar. In some wiki engines, there is even the option to use an external database, like MySQL?.

What we want to do

If we separated the database and the rest of the WikiEngine into two separate programs, then we could mix-and-match. The database has to be running on the wiki host's server, but the frontend could run anywhere, even on the client. This means that a wiki user could choose to use a MoinMoin client to interact with the CommunityWiki database, even though CommunityWiki's software is OddMuse, and even though OddMuse is what you see when you go to in your web browser.

Conditions that have to be met

Two things have to happen to allow this to occur.

  1. Right now, all this talk between the wiki engine and its database part goes on in secret. The wiki engine is the only one who ever talks to its database. We have to break the wiki engine's monopoly on the database. We have to provide an interface for anyone's program to talk to the database via the internet.
  2. In order to "mix and match", we must agree upon a standardized language for interacting with the database.

How to do it

If you think about it, the functions that a wiki's PageDatabase provides aren't specific to wikis. The PageDatabase stores a bunch of text documents. It also keeps track of the version history of those documents. It's doing the same thing as CVS. In fact, one of the programs that OddMuse is based on was "CVWiki", which actually did use CVS.

So, if we separate the PageDatabase from everything else, it'll be a CVS server. We already have programs (like CVS, and Subversion) to do this.

We want to be as interoperable as possible. Is there a standard protocol for interacting with things like CVS? There is, in fact, although as far as I can tell, it's never been fully implemented in open source. It's called "DeltaV", and it's an extension of WebDAV. WebDAV is a protocol for interacting with a database of documents over HTTP. DeltaV adds versioning features to WebDAV.

I don't know of any open-source program which claims to be a full-featured, compliant DeltaV server. But I know that Subversion claims to be close.

So, what we should do is this.

  1. Ideally we'd have a DeltaV server, but we don't, so we start with Subversion. Now we have a "backend" program to handle the database. This backend is better than the backends we currently have, because users can interact with it over the internet using (something similar to) a standardized protocol.
  2. We take the code for a WikiEngine, and we scoop out all of the database functionality. We replace this to calls to an external Subversion client. Now we have a wiki frontend which can interact with the backend.
  3. The wiki host runs both the frontend and the backend on their server.

The result

Now, a user can visit the wiki web site just like they did before. It's true that instead of using its own "database part", the WikiEngine now talks to an external Subversion server, but from the user's point of view, nothing has changed.

However, the user is also free to use other programs which interact directly with the wiki database. For example, programs already exist that let you mount a WebDAV document collection as a directory on your computer. You can edit the files in that directory, and the changes will be communicated to the server using WebDAV. So, once we do this, users will be able to interact with wiki pages as if they were text files.

Users could also edit many pages on the wiki offline, and then send their edits to the wiki all at once. A Subversion client could do this, so again, the software already exists.

There is a pattern here. By using a generic standard protocol like WebDAV, rather than a wiki-specific protocol like XmlRpcForWiki?, we can take advantage of generic software which was written without wikis in mind.

However, one could also imagine writing other programs especially for wikis. Like a wiki client.

Security: one detail

Actually, there's one more thing we'd have to do. Today, a wiki frontend is a middleman between the user and the database.

It fulfills four types of functions as the middleman:

By cutting out the middleman, we've also cut out security. For example, OddMuse doesn't permit anyone to add anything on the BannedContent list to any pages in the database. If spammers could bypass this check and access a Subversion database directly, it wouldn't matter if the OddMuse CGI frontend still checked.

Luckily, the solution is pretty simple. Subversion allows you to specify a "pre-commit hook", which is a "gatekeeper" script that is run each time someone tries to write to the database; the script gets to decide if the change is allowed or not.

Modularizing further

Separating out the database is the most important thing to modularize, so it should be done first, but I think we could benefit from cutting up the wiki frontend into even smaller modules.

In the meantime...

So, it would be great if wiki developers re-wrote their software to be more modular, and if wiki hosts installed that modular software, even though it is more complex to install. But many wiki developers won't want to do that, and even if they did, it would take forever for the new software to propagate. And many hosts won't want to bother.

There is still a way to allow users to access wikis with any clients that they want. We can build a gateway server. A server that, to the client, looks like a DeltaV, WebDAV server serving a wiki database. But, actually all this server does is translate incoming requests into HTTP calls to the real WikiEngine. The gateway would let users interact with "legacy" WikiEngines using the WebDAV protocol.

The communication with the WikiEngine part is what I'm working on with WikiGateway.

I've already written a program 1 which allows me to build a WebDAV-->WikiGateway-->wiki gateway to any existing wiki. This is how I was able to mount CommunityWiki on my hard drive as a file folder.

Unfortunately, the CGI library which handles the WebDAV part of the chain is in Python and runs rather slowly, at least on my 166 mhz server. And there's 2 links in the chain. So the latency of this approach makes it impractical for doing all my wiki reading and editing.

However, I think that if I used a C implementation of the WebDAV stuff, it may be fast enough. Also, if the upstream wiki server were a ModularWiki, I could cut out this whole gateway step, in which case I think things would be fast enough. Or maybe I just need to get a faster server. – BayleShanks

I was always reluctant to add database support to Oddmuse. Using a webservice such as the one provided by Subversion, on the other hand, sounds very interesting! I'd be interested in writing such a module for Oddmuse.

The HttpPipe? and LocalNames make it so we can break out wikitext->xhtml and publish name-url bindings as well. ;)

LocalNames is a direct product of ModularWiki & OneBigSoup thinking.

BTW the Python Twisted Web2 webserver framework now has DAV support:

See also IntComm:ModularWiki.



1. using someone else’s program for the tough stuff

Define external redirect: MySQL LogServer HttpPipe XmlRpcForWiki

EditNearLinks: WikiRefactoringBrowser WikiNames PageDatabase OddMuse WikiEngine MoinMoin