ContentRouting

“Content Routing” refers to getting information from point A, (point of creation,) to point B, (point of use, or point of integration.)

Ideally, there’d be no “content routing” to do- we’d just write or draw, wherever, and it would just be there. But in the common case that that’s not the situation (for example, the LongImageIncorporationProcess on most web forums,) then we have to perform some sort of content routing. Usually, it’s done by a mixture of manual process and mechanical automations.

AutomatedContentRouting

Here’s a diagram that shows manual (or semi-manual) content routing, and then what automated content routing looks like.

AutomatedContentRoutingImage

It is important to remember a couple of things:

LionKimbro wrote the InkscapeToOddmuse scripts, to help make it easy to route images from InkScape into OddMuse. And Lion is working on the ZergCreep (tentative title,) for more general ContentRouting, all sorts of content routing.

Content routing software isn’t very sexy. 1 You can’t take screen shots of it. It tends to be invisible. But it can make an amazing difference in how interaction takes place.

See Also

Discussion

Perhaps it’s just because I have an offline laptop while working in the bus, but I’ve been thinking a lot about Content Routing lately.

Examples:

  • I’m a software developer, working on a program. I get an error message in the program. The error message is wrong, though- it’s actually something else, something else that should be checked. I want to just edit the error message in place, and say, “go,” but it’s not that easy. The text is in a subversion server. The subversion server receives data from directories. The directories are presently on another computer, but it could be e-mailed to. One path of routing would be from display to clipboard to email to target computer to directory to subversion server.
  • I want to contribute to wiki from the bus. Text buffer to interpreter to email to email server to wiki page.
  • And the LongImageIncorporationProcess problem.

Things to think about:

  • specifying paths
  • medium’s capabilities for encoding paths
  • addressing schemes
  • subscription to addressing spaces

Prepare for a rambling reply without a point.

This sounds like a very difficult problem. First, we’d want a FineGrainedAddressing? scheme that lets you send your change/addition exactly where you want it. Next, we’d want a universal formal language for describing suggested changes and additions to any sort of thing at all. Finally, we’d want the architecture for routing the suggestion to its destination.

But the informal, non-automated system is pretty good (write a note to yourself and then fix it when you get back online).

I feel like the next step is something like what Arch is trying to do (truly decentralized VersionControl? systems that can be forked and remerged all the time). But even then you wouldn’t be able to modify the error message “in place” unless you had had the foresight to check out the source from the repository before you went offline.

Or maybe I should take this page at its word and focus on the ContentRouting part of the problem, rather than the addressing. If all you want is to get an email to yourself at the right place and time, usually you can just email yourself, and then when you get that email you’ll be “online” and every resource will be accessible; so there’s not much of a routing problem (one space/time node, the “online you”, is connected to everything; another node, “you on the bus” has limited connectivity, and so must “route” by forwarding to-do items to “online you” – but there’s always only one hop necessary, since “online you” can always access the destination directly).

But maybe you want to consider the (mostly hypothetical in my experience?) situation where there is not just “online” or “offline”, but rather a bunch of semi-connected islands – a home computer, a work computer, a laptop, and none of them can be fully controlled remotely, and some of them only receive/send email intermittently. But this isn’t THAT much of a problem as long as they do each get email sometimes; as you said, if you have an email account for each computer, you can email to-do items to yourself that you’ll receive when that computer next connects to its email.

For me this never happens, because all of my computers can be ssh’d to. So, if I’m at work, I can ssh into my home computer, and if I’m at home, I can ssh to work, and in either place, I can ssh to my sourceforge projects. So either I’m online and connected to everything, or I’m traveling with my laptop and queueing up messages that can be send when I get online.

So I guess that the “clipboard to email” part of the path has to do with FineGrainedAddressing? in my mind, and is hard to do automatically but easy to do informally, and the “email to target computer” isn’t a problem.

Heh heh heh… ;) I feel like I’ve hit paydirt.

I think…

  • …there is more distance between medium-points than we believe.
  • …we can make a “light” augmentation system.
  • …we could strike at programmatically reserved e-mail addresses.
  • …we need to understand our e-mail servers better.

Let’s not worry about getting exact targetting just yet, except where it is easy or obvious. (For instance, targeting a page on a wiki is both easy and obvious: it’s just page-name@wiki.some-domain.)

Let’s worry more about getting programs e-mail addresses, and getting their mail delivered to them.

But I’m putting the cart before the horse.

I think the key thing to notice is that there is great distance between medium-points.

It takes time to start putty or ssh, to enter the target address, to put in your username and password, and then to cd to the right directory, and then to open vi with the right file, and then to find the right paragraph, and then to enter insert mode, and then to type in the revision, and then to tell it to save the document, and then to get out of vi, and then to tell SVN or CVS to store the document, and then to give it IT’s username, and then give the password there too, and then to resolve any update problems that may have occured, and then to back out, close the session, and to shut down putty.

All that stuff takes time. And that’s what I mean when I say, “there’s distance between these points.” Point A was looking at an error message, and Point B was where you’re actually editing a version of it. And it’s in this special bubble within a bubble that you’re doing it: the particular computer, the particular check-out, etc.,.

I can imagine that, instead, you get that error message, and you send an email to my-project@svn.server.example.net. And then when you next go to svn, it tells you, “there’s an email, I’m loading it’s contents into a file.”

That gets rid of some of the traversal. We can probably do better. We can probably give a document name, and perhaps svn can prompt us somehow, that this document has such-and-such proposed change next to it.

Perhaps the error message can tell us it’s email address when it shows itself. How long to register e-mail address for the message? Can we do it programmatically?

Automation, automation. I believe it can be done, and further, that it is profitable to do so.

Why do I believe it? Because I’ve seen firsthand what a wonder removing the LongImageIncorporationProcess is, and how much a difference it made in CommunityWiki visual content.

Well, part of that story could be simplified by the right tools. Open terminal, type hostname, type screen -dr, find file, edit, save, magic key combo for version control interface, type change log, submit, kill window. This works because for the computers I use I have shell scripts that do exec ssh for me to the right host using the right username. I don’t have to enter a password because I have a ssh-agent running on my machine. On the remote machine, there’s a screen sessison with Emacs that I don’t quit, and Emacs has very good integration for files under version control…

Other than that, I’m trying to use procmail to connect mail to programs. See Email2Wiki.

I agree that we should improve this infrastructure. I also think that maybe we already have the tools to build the infrastructure and what we need is first instructions, and then packaged installations.

Doesn’t work so well for me: We’re constantly installing and wiping our systems. While I have one FC3 Linux install that is “mine,” the rest of the computers are constantly being born, smashed, revived, smashed again. We have a directory full of “install these” and we have lists of things to do, but too much gets to be too much. Jury-rigging things is okay for my one computer, but we don’t have a over-the-network-copy-and-paste yet, and really, we’re talking about spotting errors while we’re in the middle of more important things. We can spare an email with some text, we can’t spare the time it takes to locate the file in the subversion repository.

I know pageant, puttygen, and all those things. But “the right tools” alone aren’t cutting it.

I’m talking about a general problem, ContentRouting includes InkscapeToOddmuse and what not. There are hoards of places where ContentRouting is an issue. It prohibits types of contributions that could have been made. It’s just a matter of automating the ContentRouting.

Especially if our goal is to empower people; we need to make this super simplel. We can’t be lecturing people on how to “use the proper tool;” I think we instead need to make the proper tool super-simple to use.

Everyone knows e-mail, and it’s built into most of our infrastructure. Similar with bookmarklets, though not as many people know how to use them, yet. (I suspect that they will.)

You know, I was just thinking- fine grained addressing ain’t all that hard!

Just- with that error message, have a random string. And by “random string,” I mean: “Just bang out characters on your keyboard.”

When you get the error message, it says “Error ID: ERR-4958jfosi49jpAW”

Now: When you send that e-mail, send it to that random string at-subversion-dot-example-dot-com.

Send it to ERR-4958jfosi49jpAW@svn.example.com.

Then, when the subversion server receives the email, it does a grep through it’s repository for the match.

Associate the mail with the file- perhaps you had the error text in “foo.cpp”. We put the contents of the email in “foo.txt-mail-001.txt”

Bam!

So what we end up with is: The addressing system is “key words.” You can send mail to a key word. When the mail is received, a file is created next to every file including the key word.

Sanity checks are obvious: Don’t make more than 3 files, for example, …

Yes, but it’s more complicated than that. For THAT USE CASE, the right thing to do with the email might be to create a fine-grained email address for each source code file, and then when an email comes in, make it into text file right next to the source code file. But for a wiki, the email addresses might correspond to pages, not files in the filesystem, and the thing to do with incoming emails is to post them to the wiki. The problem is that each kind of application implies a different kind of object to correspond to email addresses, and a different thing to do with the email once it gets there. Hence, in general, we want some kind of hierarchial, very flexible scheme for applications to register what kind of addresses they need and a formal language to describe what to do with the incoming emails.

Of course, the first thing to do is to take the low road and just let applications register “hey, any email that comes in to ____-wiki@localhost should be handled by ME”, and then let the application developers write code to express which email addresses exist and what to do with incoming emails. Even with a formal system, this should be allowed for extensibility. And it’s easier because we don’t have to create a standard language for specifying email addresses and actions to take on emails (all we have to do is create a standard file format for the “email handling registry”, which would be a file like /etc/mailcap, and a program to read that and create the appropriate config file changes in /etc/exim.conf2. Who knows, maybe modern mail agents already have a way to do this).

In the long run, though, we’d want to take the high road and create that formal language. Because then application developers could write a line or two of email handling instructions in order to add email capabilities, instead of writing special mail processing scripts, which few people would bother to do.

In summary, I agree with Lion that the problem is:

It takes time to start putty or ssh, to enter the target address, to put in your username and password, and then to cd to the right directory, and then to open vi with the right file, and then to find the right paragraph, and then to enter insert mode, and then to type in the revision, and then to tell it to save the document, and then to get out of vi, and then to tell SVN or CVS to store the document, and then to give it IT’s username, and then give the password there too, and then to resolve any update problems that may have occured, and then to back out, close the session, and to shut down putty.

And the thing to do now is: learn more about programmatic control of how email is handled.

But I think the final solution to this problem is a very long way off.

p.s. procmail syntax stinks. Is there some sort of more readable language that can be compiled into procmail?

Since I had a hard time every single time I tried to configure the local mail transfer agent – be it sendmail, exim, or something else – I have decided that I will avoid the technology if at all possible.

While fetchmail is ugly and arcane, I can manage to write a simple rule. In this case I used the following:

:0 fw
* ^Subject: CW:
| post-it

The rest is in a script. And the content of the script in this case (no handling of encoding and attachments):

#!/usr/bin/perl

my $script = 'http://www.emacswiki.org/cw';

sub main {
  $/ = undef; # slurp files
  my $data = <STDIN>;
  ($headers, $body) = split(/\n\n/, $data);
  $headers =~ /^Subject: CW:(.*)/m;
  $subject = $1;
  die "no subject\n" unless $subject;
  die "no body\n" unless $body;
  open (F, "| wikiappend '$script' '$subject'") or die "cannot call wikiappend\n";
  print F $body;
  close F;
}

main();

It uses Oddmuse:wikiappend.

It is probably a security risk, too. :(

I’m breaking the old summary, but think it may be useful when we finally rework this mess: “Three situations calling for content routing. Ramble. “The distance between medium points.” Tools to reduce that distance, are they enough? Use procmail to connect mail to programs. Using key words as fine grained addressing. Procmail is butt-ugly but works. Sendmail never does.”

Here’s one:

HavocPennington? writes about version control.

He says:

 If someone submits a patch, I should be able to just click
 "apply to HEAD" and it's done. It shouldn't be about getting
 a clean checkout, downloading patch from bugzilla to local
 drive, applying patch (figuring out which patch options the
 person used), creating the commit message, etc. Should be
 just one click.

Heh heh, on CommunityProgrammableWiki it’ll be that easy (for different reasons).

I just saw "Atom as a Case Study," and it seems one of their goals for the Atom Publishing Protocol is to ease ContentRouting.

Aza Raskin talks about things that are more complicated than they should on this Google Video. He touches the subject creating web pages too.

Footnotes:

1. For more on multi-touch: http://mrl.nyu.edu/~jhan/ftirtouch/
2. on my new system, I guess it’s /etc/exim4/conf.d/

Define external redirect: FineGrainedAddressing VersionControl HavocPennington

EditNearLinks: ChrisDent

Languages: