CommunityRepository

The proposal is to establish a Community Content Repository.

CommunityRepositoryImage

Proposed Directory Structure

 projects/
   bank/
 groups/
   cf/             -- coforum
   crao/           -- CraoWiki
   cw/             -- CommunityWiki
     pages/
     images/
     renderer/
   mb/             -- MeatballWiki
   ww/             -- wikiwiki.de
 individuals/
   alex/
   christophe/
   david/
   emile/
   hans/
   helmut/
   lion/
   mattis/
   mark/
   sam/
   sigi/
   ted/
   and so on...
 admin/
   logs/
   records/
   taoriver/

(This isn’t to say who’s in, or even wants to be in; Just to feel what could be there.)

Why?

Table

See Also

CategoryRevisionControlBasedWiki

Discussion

Outsiders

The question is: “What about outsiders?”

One of our favorite things about wiki is the ability of total strangers to come by, and start contributing immediately.

We owe a very large portion of what we have, to this single feature alone.

Were we to substantially commit to the CommunityRepository, this would obviously make things difficult, in this respect.

One possible answer is to make the renderer render out HTML form pages as well.

When we render the website, we can have the renderer render out pages with HTML forms that have the text of the page in it by default, and that people can hit “submit” on. When they hit submit, they are posting into a submission queue of some sort. We can then write scripts to pull from the submission queue (somehow,) and integrate with the repository.

Or something.

There should be some way of getting content in.

What about allowing users to submit their public key via web form to give them access to the repository?

That’s awesome!

My only concern is security: I don’t want to just hand out shell access carte blanc. There have historically been a number of shell account exploits used on FreeBSD. I would be too easy a target.

I do think accepting public keys is the way to go, though: No trouble distributing passwords.

Since I’m still bootstrapping my PKI knowledge I may be wrong, but I currently believe that the Public part of a key pair is exactly that … Public. If so, I fail to see how it could be easily used as part of a login security scheme. Can someone enlighten me further on this?

Sure!

Hans, you paste your public key here. (At no risk to yourself.)

I want to let you log in as “repository” on “taoriver.net”. I past your public key into /home/repository/.ssh/authorized_hosts.

Now you can log in. It works like so.

Your computer contacts my computer. It says, “Hi! I’m Hans!” My computer says, “Hans, I’ve got a challenge for you. A challenge that ONLY YOU can answer.” My computer encrypts a secret message by the public key you gave me. (I’m not sure how it knows which of the keys is the one you can answer. But somehow, this works.) It sends that encrypted message to you.

Unless you’ve given your private key to someone, your computer is the only computer that can decrypt the message. It sends back the decrypted message.

My computer goes, “Oh, Hans, it really is you! Come right in!”

Thanks, Lion.

I was missing the part about the challenge question that you would send, encrypted with my own Public Key. Now its obvious.

For your information, the way the key pairs work is that the Public and Private keys are related via a common Modulus. This allows the receiving Key to decrypt messages from the sending Key. This mechanism makes it possible for the Public key to be used by any sender and the owner of the matching Private key to always be able to read the message sent to him by anyone using the matching Public key. Its a bit of a “slippery concept” according to one of my associates who does a lot of work in this area. On the basis of several false startes I’ve taken at this over the years, I’m inclined to agree with him. If it becomes important to you to have this clarified further, I could take a shot at creating a “…For Dummies” version since I think I’m qualified as a Dummy on this subject.

By the way, I’ve been told that PGP has some attractive features that I should look into. Do you (or anyone else, for that matter) have any experience with it?

Lion, you can set up GnuPG? and Procmail to automatically apply OpenPGP?-signed/encrypted darcs patches sent via email with “darcs send”. There are some instructions for setting it up in the darcs manual: "Sending signed patches by email".

Hans, I think something like a “Public Key Cryptography For Dummies” might be very useful. Would Wikibooks: Cryptography or the Cryptodox wiki be a more appropriate place?

p.s.: I wrote “Cryptodox: Vote” because I thought “end-to-end auditable voting systems” were completely impossible, and I was astonished to discover that they were possible after all. (I suspect I don’t do a very good job explaining what little I do know …)

David…

I took a quick look at the links you’ve provided and I think given the information that they provide and some of the more recent developments that have happened since the start of the year, it may indeed be good to try to create a “for dummies…” page on the uses of Certificates and Encryption within the CommunityWiki contexts. I’ll think about this further over the next few datys and then post again.

I suspect that the systems used in “financial cryptography” (Wiki:FinancialCryptography) are very similar. That may be relevant to the CommunityWikiBank.

Recent Changes

It’s unclear what RecentChanges would look like. Or what we would want it to look like.

Do you just take the darcs changelog, and make RecentChanges out of that? Or do we want to keep a record by hand as we commit changes? Or..?

And just how much should go into a darcs changeset?

Front Page

I talked with BayleShanks on the phone, and he liked the idea a lot.

I’ve also set up ShawnKilburn, over the phone.

I believe I’ll get LukeStanley set up Tuesday.

Other

So far, I’ve:

  • installed darcs on FC5, FreeBSD (TaoRiver?)
  • taught myself to use darcs
  • created basic layout, vanilla renderer
  • user “repository” on taoriver.net, setup darcs repository
  • hooked renderer output to Apache

Here are the two links so far:

In fact, though, the first repository get, you want to use:

 darcs get repository@taoriver.net:/home/repository/base

From there, it’s:

 darcs pull  # update from repository
 darcs push  # save changes to repository

To do this, you want to key yourself on the repository server. I’ll be distributing the repository password by SMS or phone call or encrypted mail.

You’ll need to generate SSH keys, and remember your password to them. I’ll help everyone who needs it. You’ll want to key up repository@taoriver.net. Or you can paste your public keys here, and I’ll put them on the server.

For the less technical here, this will be a big learning opportunity.

LukeStanley & ShawnKilburn have both expressed in the system. My goal is to get them set up by Tuesday night.

You can see “where I’m at” by looking at my repository notes file.

I did some reading up on DARCS, because I was not previously familiar with it. I have to say that the idea of “distributed revision control” is amazing.

I’ll be interested to see how this evolves, and experiment with it. I am a novice at many programming tasks, such as using emacs, revision control systems, etc. But that has never stopped me before.

Yah! What operating system are you using?

So far, I’ve set this up on Mac OS X, Linux, FreeBSD, and Windows, so I’m fairly comfortable doing it now.

I am currently running Mac OS X 10.3.9

Great! We just got ShawnKilburn up and running, on Mac OS X.

The install steps, if I recall correctly, were:

  • (1) generate an ssh key (run ssh-genkey (I believe,) and then post the ~/.ssh/id_dsa.pub here, for me to key the repository)
  • install darcs – which I believe required installing something else, first, …
  • python ez_setup.py PyYAML – which required downloading ez_setup.py from the PEAK website (and must be run as su)
  • then do a:
 ssh-agent $SHELL
 ssh-addkey
 darcs get repository@taoriver.net:/home/repository/base

We need to make a full sheet on installation though; on CommunityRepositoryInstallation?. But until we’re there, ask questions here, or call me on the phone as you’re working.

You’ll also want to install InkScape. To run it, start X11 first.

To update the darcs repository:

 darcs pull

To save edits to the repository:

 darcs add (filename or directory here)  # do this the 1st time you introduce a new file, only
 darcs record -am "commit message"  # the 1st time ever, you'll be prompted for your password
 darcs push

I just moved my outliner notes for CommunityWiki into the repository, and set up a basic renderer for it.

You can see the output at: http://repository.taoriver.net/lion_ideas.html

How is it rendered?

It is rendered using emo2html.pl, a script I saw on EmacsWiki:EmacsOutlineToHtml.

It’s a primitive script. I’d like to extend it, to let me anchor parts of the outline, and perhaps even compile a LocalNames namespace description from it.

But for now, I just have to tell you what to look at: If you scroll down to II. CommunityWiki, you’ll see a number of thoughts I’m holding in queue for the CommunityWiki, along with some of the “filling out” I’ve done so far in my mind.

II.A.4. describe a concept that I’m calling the “Web Bottleneck,” and that I see us trying to work around. ThomasKalka, you may be interested in that.

So, got a little off track with the ProcessNetworks? and business discussion on CommunityWikiBusinessDiscussion. None the less, still interested in this. I think Lion had an interesting prototype here.

Before I really dive into this here, I think i am first going to start up a page here on SemanticWiki, and pull some content from different discussions here into that page. Because, SemanticWiki ideas are related to the way that I am thinking about CommunityRepository

Hey, this looks great. How do I turn this into a truly distributed wiki? What do I need to do on my server to get it synchronized with Lion’s server?

Someone seems to be way ahead of me:

Brock Wilcox says “Oddmuse Editfile is a simple oddmuse plugin which will let me, among other things, put some of my source code directly into wikiland. Changes from the wiki will be checked into darcs for review.”[1]

Later, Brock Wilcox says “I got the wiki-to-darcs (both-way) gateway going.”[2]

TheSheep also had an idea for a DistributedWiki (or “P2PWiki”) maybe he will weigh in on that?

I think that, overall, directly employing the infrastructure of code revision systems to manage wiki data is bad WikiEngine design. If you look around on WardsWiki, you’ll see examples where people used CVS for similar purposes when Wiki was first taking off.

Instead, it might be good to borrow the infrastructure, code similar ideas into your own wiki engine.

Also, for DistributedWiki, it is possible to synchronize http://moinmoin.wikiwikiweb.de/WikiSyncronisation is possible in MoinMoin

“I think that, overall, directly employing the infrastructure of code revision systems to manage wiki data is bad WikiEngine design.”

Why? I always wondered why wikis moved away from using code revision backends, and I’ve dreamed of moving them back. But I agree that the fact that a bunch of wiki engines used to be that way, and now they aren’t (more correctly, the ‘most popular’ engines aren’t, afaik) means that there was probably some problem with that method. But what is it? If you know, please tell me!

Incidentally, OddMuse came from UseMod which came from AtisWiki? which can apparently can use a CVS backend. AtisWiki? came from CvWiki? which requires CVS. However, in this lineage, the reason for dropping CVS seems clear: a primary feature of UseMod and OddMuse is ease of installation, part of which is the minimization of dependencies. I understand this reason. But why is it that other wiki engines, which don’t mind having heavier dependencies (for instance, those which require an SQL server) don’t just use code revision systems?

I guess that mySQL is already installed on most hosting servers, and the database setup can be automated – so users don’t really have to deal with it. Also, the idea of a database is widespread enough to not be that scary. Revision control systems are poorly understood and usually remind of something exceedingly complicated like CVS or Subversion.

Another potential reason that comes to my mind is that programmers who write web applications instinctively cringe from calling external binaries: because it’s easy to mess up and create a gaping security hole.

Personally I’m thinking about using Mercurial for a back end: I can call it from python directly, as it itself is written in python, and it has excellent support for synchronizing repositories – actually that’s its main mode of operation.

Bayle, I think it’s a bad design, partly because the solutions that emerged in many wiki engines have way less “overhead” server resource and time costs, easier to set up and maintain. And, the database that is used is generated directly by the code, it’s creation is integrated into the script. The script also tracks revisions, and employs diff. Over all, for most wiki applications, that is a better design from what I can see. For people like us who want to explore and experiment, I suppose there really is not reason not to set up a wiki that copies to revision control, and see what advantages could be gained from that exploration. Radomir, I will look into Mercurial, thanks!

So, to rephrase what you two are saying to make sure I understand, the downsides of using a code revision control system as a backend are:

  • harder to set up, to maintain
  • executing external binaries can lead to security holes if you don’t properly sanitize user input
  • “glue code” to pass information between the wiki and the revision control backend leads to an inelegant program
  • the functionality provided by (and hence resource costs demanded by) code revision control systems are overkill for what is needed
  • external databases share some of the above disadvantages, but everyone has more experience with them (leading to easier setup, less likelihood of making a security bug, more elegant glue code) so the disadvantages are attenuated

Did I get it right/miss anything?

Well, there can be one more factor, the same that often makes people start writing a wiki engine at all (it was like that for me at first): “I want my own.” :)

Bayle, I think you got what I was saying mostly right. I am basically conjecturing that some wiki engine coders selected to bypass using revision control, at least in part because creating one installable script made the likelihood of people downloading and using their wiki engine higher. I think you could actually ask people like Alex, Helmut, Radomir, CliffordAdams, and some of the people on WardsWiki, and I am sure you’d get some other reasons. I think it would have been a turnoff for some people over the years to ask them to set up both a revision control system, and a wiki script, and get it all working correctly. I think the revision control idea appealed more to programmers, and people who were experimenting with wiki engine function, than with folks who just wanted to download and admin a wiki.

So, it’s exciting to me to explore revision control wikis, but I think we are a small niche in the realm of people interested in wiki. Although, if the revision control and wiki were made to be one “package” this could make it’s appeal more widespread.

What do we think the “pros”, or advantages of revision control wiki are? I can see some in Lion’s diagram above, but what else, if anything?

Let me see.


With a revision control system you get a lot of “features” for free (at least if you bother to write the interface for them), depending on the actual system used:

On the other hand, relative databases like mySQL usually give you:

(please add to/edit this list directly)

I’m in the process of writing a wiki-like system right now. Here is why I’ve avoided databases and versioning systems.

  • “I’m just keeping some files in some directories.”
  • “It’s an easy enough way to get started.”
  • “No dependencies.”
    • That is, the leap from 0 to 1 dependency is bigger than the leap from 1 to 2 dependencies.

Are these good reasons? Maybe, maybe not.

Irritating things about using external tools:

  • They’re dependencies. Yuuuuucky yucky yucky. Automatically, a bunch of people won’t try the tool.
  • The “glue code” that Radomir mentioned– it’s not just ugly, it’s also a security caution.
    • bash is the paragon of cli glue tools – but I wouldn’t use it in an insecure environment!
    • the web is an insecure environment, and “they” would love to crack your computer, or the computers of one of the people who trust you enough to trust your code
    • escaping can be damn irritating code work
      • “are you sure you got all the edge cases?”

I’d say that ease of installation is a very high priority for a lot of people.

There is nothing easier than, “Download this one file, and run it.”

Setting configuaration variables within a program takes you down a couple notches.

Specially configuring Apache takes you down several more.

Requiring MySQL? and other databases, and you’re basically hosed: The person needs to be really motivated, to install your program.

Radomir said something that gives me pause, though: indexing.

If MySQL? does full text indexing, that’s something really cool, and worth thinking about. That said, I still wonder, “Can’t I just roll my own?” Even if I do use it’s indexing capabilities, can I do so securely? Are there was of hiding commands within information requests?

On one hand, escaping everyhting properly all by yourself is pretty hard and generally a slippery ground:

xkcd

Then again, most programming languages out there come with ready libraries for using the databases, and these come with pretty throughoughly tested “glue” code already. I think that Python, PHP and Perl all have them in the standard library, so it’s not much of a dependency…

As for installation – you can always ship the dependencies with your application in a single package, if the licence allows. Windows users will not try andything anyways unless it’s a working EXE file.

I’m trying to imagine a tarfile that includes (A) MySQL? for Windows, (B) MySQL? with configuration options and tools for 15 Linuxi, (C) MySQL? for Macintosh, and then (D) my little dinky 35k Python program.

Am I just complaining, that I have to figure out how to make a Windows .msi/.exe deployment program, that will install MySQL?, my program, and then configure them together properly?

Is there some easy Python command for making it do that, that I just haven’t found yet?!?

:)

I also genuinely don’t know about a Python module that securely isolates you from SQL command strings. You can talk with the databases just fine, but you’re still passing crafted SQL strings, is my understanding.

Writing my own “files in folders” database looks all the more attractive…

Yep; sqlite works. :) I’ve used that before.

Perhaps we have found SamRose’s answer: “The reason why, is complexity in installation. When a database is good and part of the system from the beginning, it will be used.”

Then again: There’s something to be said for just double clicking on text files.

You can still have simple text files while using a revision control system – after all, you need to have revisions of something. And the diffs are also just text files, mind you. They might have weird names because of the way that the revision control system works, but well, nobody’s perfect. In the case of absence of the required revision control system, you can still have a working wiki, just without the access to history – which might be enough for a locally edited clone.

I like the idea of distributed wiki (sorry Maybe I’m a little bit biased on that subject). The diagram looks nice to describe all the output and potential resulting use. I think there is an important concept missing in the diagram where distributed SCMs like git, mercurial or darcs are very good. The branching is a key element in distributed “SCM” and that’s introducing a great extension (still not really used) for editing wikis. For example, it would be more easy to fork pages or a part of a wiki, contribute to it with a small group and import (merge) back the contribution into the original wiki. Branching and merging are now really working in SCMs. Would it be possible to update the diagram to show the branching/merging of the repository(ies) ? or in other words to show the decentralization of the wiki contribution/repository ?

Define external redirect: CommunityRepositoryInstallation MySQL GnuPG TaoRiver ProcessNetworks CvWiki OpenPGP AtisWiki

EditNearLinks: CliffordAdams UseMod WardsWiki FreeBSD MoinMoin WikiEngine

Languages: