DataMesh

Abstract: A network for routing data from any point to any other point (ContentRouting) is proposed, that can automatically handle the logistics of ftp’ing, ssh’ing, addressing and changing parts of files, across heterogeneous networks that are active & functional only intermittently.

A “Data Mesh” is proposed here to mean something like a MeshNetwork, supporting something like a DataCloud?.

A MeshNetwork is a network that performs OpportunisticProblemSolving?: It looks around for any available connections, and if it finds one, wires itself in. It’ll route for others, as well.

A data cloud, in the literature, is a hypothetical “cloud of data” that (somehow) follows you around. So, you go to your friends house, and there you’re MP3’s are: accessible from your friends computer. If, wherever you go, you have access to all your data, or all the data that’s important to you in that context, then you could be said to be “in your data cloud,” or you could say, “Your data cloud is following you around.”

A DataMesh is an idea that’s something like these two other ideas. It should be capable of filling some of the roles that the data cloud ask for. And it works opportunistically, like the mesh network. I’m considering calling it a “CyberneticMesh?,” because conceivably, it’s not just data, but executables as well.

The idea, briefly, is to opportunisticly route data.

Example Problem

Suppose you have a system like so:

This is actually fairly typical. People have access to all sorts of computers, servers hosting in all sorts of different locations, all kinds of different stuff running on them. Some are servers, some are not.

What you want, is to be able to communicate messages & do things, independent of where you happen to be, and whether your Internet connection happens to be up.

There are many things that can go wrong:

This puts you squarely in the domain of a ContentRouting hell.

Then, the next morning, after an evening of reminding myself: “Oh, remember, tomorrow, to upload that file to CommunityWiki…”

Very frankly speaking, this is utterly absurd.

So absurd, in fact, it’s easier to just not post. Several CommunityWiki posts have been stillborn due to this logic, or something like it. It just doesn’t reach high enough priority to justify the costs. (see also: SmallChangesYieldGreatChanges, section on “priority.”)

How the Data Mesh Solves It

Now, let’s go back to the beginning of the scenario: You’re at home, on your Windows computer, and you realize: “Oh, shoot, I can’t connect to the Internet.”

Instead of going through that big mess, you put the text file in a special directory. Most likely, you’re going through the following process:

(If you wanted to do it extra cool, you could have a file naming convention, or put a special set of characters at the top of your file, such that a program that either periodically watches the system, or has a constant running watch of new files added, could pick up the file, and collect it over to wherever it belongs.)

From there, you don’t worry about it.

A program (perhaps the ZergCreep) picks it up later on, and discovers that the WLAN connection to the FreeBSD computer is open. It promptly uploads the file. The file still isn’t “delivered,” but it’s out there.

Then the laptop, next time it’s on and connected, goes checks the FreeBSD computer. “Message for you, sir!” It picks up the file.

Some time later, I go to work. It detects now that it’s connected to the Internet. “Now’s my chance!” It checks first to make sure it’s not already been uploaded somewhat recently. No? Okay! Upload it!

Now, it distributes a note: “Job’s finished.” It distributes the small note to everyone it knows, and can reach. (The various servers out there.)

When the laptop gets home, it further distributes the “Job’s finished” notes. Computers erase the file that they no longer need.

So, the entire process of data routing happened by a series of piggybacks.

Odds & Ends

The process is not infalliable. A message might never get deleted, and timeout. Or a block of data might be too big to pass through a small point.

You might get double posting, triple posting, or no posting.

See Also

Discussion

I originally had this in ZergCreep, but it’s a significant enough concept on it’s own, and I don’t know if I’ll ever get around to implementing it in the ZergCreep.

Define external redirect: OpportunisticProblemSolving CyberneticMesh DataCloud

EditNearLinks: FreeBSD

Languages: