ZergCreep

The “Zerg Creep” is a Python module written by LionKimbro that aims to make a lot of ContentRouting problems very easy to solve.

It automates the shipment of content from one location to another.

For example, say you want to add some short piece of text to a wiki page in the middle of working on something else. Right now you have to drop down to the desktop, start a web browser, wait for it to boot, go to web page, navigate to the place I want to edit, hit the edit button, find the proper place to put the thing, put the thing there, hit the save button, confirm it worked. With ZergCreep, you will be able to just open a file, write a line, save it, and then go back to your work. Later, ZergCreep will look at that file, see that you have some text in there “addressed to the wiki”, and it will “route” that text to the wiki.

Another example: you want to send a file from the computer you are on to another computer. You don’t want to have to SSH somewhere, cd down to some directory, and put a file there (maybe you aren’t online right now). You just want to plop down a file, have the computer pick it up for you, and transport it there.

Another example: you have an SVG that you want to post to CommunityWiki. You just tell ZergCreep to send that file to CommunityWiki. ZergCreep takes care of constructing a temporary directory, telling Inkscape to render a PNG into it, and then uploding them both the SVG and the PNG.

In the longer run, we’d like to see the ZergCreep be adapted to not just content from computer to computer, but also make it more convenient to specify where in particular the content goes on the target computer (FineGrainedAddressing?). For instance, perhaps you could leave StickyPoints? in CommunityWiki as you write an article. Then in Inkscape, if you say “Save to the CommunityWiki,” Inkscape would show you all of the sticky points that you have left open. Then binding the picture to the place is a snap!

Further, Lion would like to see it grow to carry content across several intermittant links; a “DataMesh.”

Example Expressions

Example expressions in the Creep include:

 community_wiki.receive(ImageFile("C:\\FiveWorldsMandala.svg"))

It takes care of stripping the path, the .svg, constructing a temporary directory, telling Inkscape to render a PNG into it, and then uploding them all.

You could just as well, if you liked:

 Folder("C:\\Documents and Settings\\lion\\records\\").receive(ImageFile("C:\\FiveWorldsMandala.svg"))

…to keep a backup.

Or you can:

 community_wiki.receive(TextFile("C:\\SomePage.txt"))

Again, it takes care of all the logistics.

I’m presently adding in memory, so that you can do things like:

 for_communitywiki = "C:\\forcw\\"
 communitywiki_records = "C:\\fromcw\\"
 def process_instructions(line=None, **other):
     ...
 tally_list = [(WHEN, new_file_in(for_communitywiki), send_to(community_wiki)),
               (WHEN, new_line_in_file("C:\\instructions.txt"),
                      process_instructions),
               (WHEN, new_entry_on(community_wiki),
                      send_to(communitywiki_records)]

Everything except the last there works, right now. The last one, I’m about to work on, right after I finish the RSS memory thingamajig.

An Interesting "Check-List" User Interface

One thing I really want to do, is make it so that you can keep a file like so:

 Blah blah blah.
 Blah blah blah blah blah.
 [ ] TAG "http://example.net/" example website paragon
 [ ] TAG "CW:FiveWorldsMandala" imagination theimaginaries
 [ ] CACHE "http://example.net/foo/bar/baz"
 [ ] MAIL "kitty@example.net" "Is it okay if I eat the chicken that was in the fridge?"
 [ ] SMS "206.427.2545" "Call me, ASAP."

When the tally’s run, it looks through the files, and tries to do whatever it can do. It leaves the file looking like this.

 [X] TAG "http://example.net/" example website paragon
 [X] TAG "CW:FiveWorldsMandala" imagination theimaginaries
 [E] CACHE "http://example.net/foo/bar/baz"
     404 Not found.
 [ ] MAIL "kitty@example.net" "Is it okay if I eat the chicken that was in the fridge?"
     Could not communicate with SMTP server. Repeating attempts.
 [X] SMS "206.427.2545" "Call me, ASAP."

Whatever it X’ed, is now done. The “E” means it was an error. The blank is something it’s still going to keep working on.

If you just weren’t connected to the web, you’d just have a bunch of blanks, with “couldn’t communicate yet.”

But, I also want to do things like:

 [ ] BROADCAST Now's not a good time to call; I'm busy on a project, and don't have any time.

That message would be distributed to:

I’m not sure how the error reporting (“X”‘s and “E”‘s) on that one would work; perhaps it would just open up sections underneath it, for the particular mediums.

 [ ] BROADCAST Now's not a good time to call; I'm busy on a project, and don't have any time.
 [X] REPORT: blog received
 [X] REPORT: CommunityWiki received
 [ ] REPORT: cannot communicate with website server
 [ ] REPORT: cannot communicate with presence device

Design Goals

The goal is not just to make these things possible, the goal is to make them easy.

The interface for the programmer is focused then on:

So for example, when a text file is delivered to an OddMuse instance, and it’s wondering: “what should I call this,” then it checks if the name is CamelCase, and if it is, says, “well, that’s it’s name.” That’s an example of the assumption end of this.

The resiliance end is about error reporting, and making sure things don’t just die. This is about stability in the face of the number of things that can go wrong.

The “intuition” part is about the user interface for the programmer: how you write the codes to configure these things.

I’ve been designing the API by writing out what I think should be sensible to say in English, and then asking myself, “What language can I build to support this?”

I found myself saying “When…” a lot. But that’s not a construct in most computer programming languages, including Python. (I know that several languages use “when” as a control structure type thingie, like “if” or “while,” but that’s not what I’m talking about- I’m talking in the sense of an event system.) So, I’m writing a system for supporting those things.

I’ve also found that a lot of the things we want to say, are based in the language of change. “When you see a new file in this directory…” “When you see a new line on that file…” “When that file changes, …” “When you get an email, …”

You can also see that configuration is a very basic part of this as well; the program demands configuration. Fortunately, I’m not resorting to any sort of configuration file system, or anything like that; It’s all just straight python.

 community_wiki = OddMuse("http://www.communitywiki.org/")
 community_wiki.default_user = "LionKimbro"

…and so on, and so forth.

Between Environments & Computers

ContentRouting can work in multiple spheres:

By “between environments,” I mean: “I’m programming in EMACS. I don’t want to have to drop down to the desktop, start a web browser, wait for it to boot, go to web page, navigate to the place I want to edit, hit the edit button, find the proper place to put the thing, put the thing there, hit the save button, confirm it worked. I just want to go to a buffer, write a line, save it, and then go back to my work, without having left EMACS at all.”

By “between computers,” I mean: “I don’t want to have to SSH somewhere, go find some directory somewhere, to put a file there. I just want to plop down a file, and have the computer pick it up for me, and transport it there and do whatever with it there for me.

I want to say: “On one of my computers, I have this file, ‘foo.txt’. Bring it here to me, into this directory, whenever you can go get it.”

I want to say: “Make an index of all the files on all my computers, and replicate said index on all my computers.” (Warning: Said indexes can get pretty big… You might be surprised how large directory listings can be..!)

I want to say: “I want this thing to run on that account on that server, once every 2 hours,” even though I don’t have cron access on that server, even though I don’t have the ability to make long running background jobs there. If I can ssh in there, that should be enough.

Ambitions?

This type of thinking heads into more of a DataMesh-like territory. ZergCreep doesn’t presently do that, and I don’t think it’s very likely that it will get there. I suspect that before I ever get there, somebody else will make a beautiful system that does that, already. Or, that I’ll find the code that already exists today, and does that job wonderfully.

I’m eager to get back to LocalNames, after all. But this has been so immediately useful, I just cannot deny working on it.

The Name: "Zerg Creep"

Zerg Creep is my pet name for a project I’m working on. Other names I’m tossing around are “The Creep,” or “Creep Colony,” or, switching over to the side of the Protoss: “Cybernetic Organism.”

The name comes from in Starcraft, where the Zerg set up these organic, living bases. Wherever they set up a home base, a purple goo emanates out, and covers everything nearby. You can only build on top of the goo. There are these units (“Creep Colonies”) that allow you to extend the goo out in different directions. And if you have several bases, with goo-less land in between, you can connect them with these cave-like wormholes.

The goo is called: “the creep.”

The idea here is that you would install a Zerg Creep Colony on each of your computers, configure it a little for it’s local space, connections, who it trusts, etc., and then there you are.

Perhaps the DataMesh software would be called “CreepColony?.” ZergCreep, then, is the name of the module it uses to do stuff.

Or, not. These names are likely trademarked. Perhaps I should just call it: “The Creep.” Or more boring but descriptive names, like “Content Router” and “Data Mesh Node.”

See Also

Discussion

My big motivator was the realization that, even when I have a beautiful wonderful TabletPC?, the ContentRouting problem is still there, it’s still deeply entrenched!

In fact, I was talking with RaymondKing? a few days ago.

We sat down, and I said, “What I’m really eager for, what I really want, what I’m really excited about, is the coming of the Tablet PC.”

I said: “Come August. In August this year, 2006, I’m going to by myself a Tablet PC. I’ve been planning this since my last birthday. Amber knows. In the finances, it’s all tucked away. There it is. In August, I’m getting myself a Tablet PC.”

I was getting all excited about it, just imagining it.

But then RaymondKing?, he said: “Oh. I have a Tablet. I hardly ever use it.”

I said, “Ah..! And, why is that?”

He said: “All you end up with is a bunch of JPEG’s sitting on your hard drive.”

And I said: “Yes! I know! And that’s exactly the problem I’m so concerned about!”

I think he thinks I just made that up on the spot. Little does he know, I’ve been obsessing over this very problem, here on CommunityWiki, and in my private thoughts. Note that it is also at the heart of LocalNames, as well.

I’ve thought of all kinds of crazy things. “What if I could leave sticky points in CommunityWiki, when I’m writing an article. And in Inkscape, when I say “Save to the CommunityWiki,” have it show me all the sticky points that I’d left open, as wanted pages? Then binding the picture to the place is a snap!

I knew exactly what he was going to say. Yes: The problem is, you end up with a bunch of JPEG’s sitting on your hard drive.

Fortunately, unlike chip design, unlike computer assembly, software is something that I can do something about. And we’re not talking brain surgery here; We’re just talking moving files around.

But to what dramatic effect!

“The creep” is a purple substance that the Zerg emit from their “buildings,” (scare quotes, because the buildings are entirely organic,) which they are confined (for the most part) to building upon.

It refers to the “mesh” created by intermittantly connected computers.

So, ZergCreep is like an event system running backwards? The user describes an “event” that they want to happen (whenever it gets a chance to), and then ZergCreep propagates that event to the place where that event “happens”. (rather than a traditional event notification system, where something “happens” at the server, and then the next time there’s a chance, users who were listening get a message about it).

Sticky points are equivalent to the areas on the server that you could “listen to” in an event system.

Or, it’s like synchronization except that you don’t want to have to maintan a local copy of the entire server; you just want to write a “diff”, i.e. instructions for changing something, and then apply that diff to the server the next time you can.

That stuff about building and replicating indexes and psuedo-cron jobs can be seen as a generalization of this in which the “reverse events” or “diffs” are not just pre-specified changes to datastores, but rather pieces of code that are run on the server.

Propose to FrontPage this entry (after a few days of rewriting the initial intro).

Yay! You like it!

  • Give me time to make my diagrams.
  • What specificly (besides links to Zerg & Protoss?) you want changed from the intro, as it stands.

Yeah, you guessed it, I want the name discussion moved down, and I’d like some examples in the intro too. Anyway we can edit it after you make your diagrams.

I made my diagram, and it turned out to be about ContentRouting, rather than the Zerg creep!

So I think the thing to do then, is just to clean up the page a bit, and post it to the FrontPage.

That, and upload the latest of the code so far.

Okay, Bayle- if you could check this over, make adjustments as you think are sensible, …

Define external redirect: TabletPC FineGrainedAddressing StickyPoints RaymondKing CreepColony

EditNearLinks: CamelCase

Languages: