Centralized versus Distributed Knowledge

The Central Information system is envisioned at a repository of information and experience, from library books for general reading, a computer database to map topics and indices of the materials available, and interactive computer programs and instructional video “how-to's” to relate what we have learned, in sufficient detail that the system can almost become self-instructive.
Post Reply
User avatar
LoneBear
Legatus Legionis
Legatus Legionis
Posts: 3578
Joined: Thu Jul 22, 2004 12:38 am
Location: Utah
Contact:

Centralized versus Distributed Knowledge

Post by LoneBear » Wed Sep 23, 2009 6:30 pm

One of the big problems that the Library at Alexandria had was that it was a bit too "centralized", and when it got taken out, all that information was lost forever. In the modern world, the Wikipedia is basically the same thing, though they at least have backups--yet, all the information is kept in one place, save what everyone copies from it.

I'd like to set up the CI system so that information is NOT centralized, but distributed over a number of "cloud" organizations, in a fashion similar to DHT (Dynamic Hash Table) protocol. If you know about the a particular bit of info, but don't have the details, you get it from someone who does. Kind of a "pass the buck" system, but it removes the problems with centralization.

The second issue is redundancy... every bit of information should also be someplace else, but not necessarily in one spot. Similar to downloading a torrent... you get bits from hundreds of sites, to assemble your file. That way, if something is not reachable, you still have a good chance of getting the entire dataset, though it takes a little longer.

I'd be interested in ideas on how to implement these concepts.

User avatar
Tulan
Cellarius
Cellarius
Posts: 453
Joined: Wed Aug 31, 2005 8:04 pm
Location: Austin, TX
Contact:

Re: Centralized versus Distributed Knowledge

Post by Tulan » Sun Sep 27, 2009 3:20 pm

The Freenet project has an interesting implementation of completely anonymizing information over all participating clients; the security of such a system may be an interesting proposition to consider.

(Note: Freenet uses a distributed filesystem, that's why I bring it up, even though it is primarily used for anonymous information)

Freenet does have its shortcomings, but, could be interesting.

4store (the RDF Triple Store/Graph Database I mentioned in the post on Database design) is built to be a distributed system. Erlang, the programming language, is uniquely designed to handle concurrency, distribution, and fault tolerance (something to consider if data were distributed).

Does the distribution look something like - across participating members or across institutions such as the Amazon S3 service? Or a mix of both? I personally like the idea of setting up something like Freenet but serving entirely as a data store that members participate in. A bit like the folding at home project in some ways, except more for data store than crunching.
Ah, you seek meaning? Then listen to the music, not the song. - Kosh Naranek

User avatar
LoneBear
Legatus Legionis
Legatus Legionis
Posts: 3578
Joined: Thu Jul 22, 2004 12:38 am
Location: Utah
Contact:

Re: Centralized versus Distributed Knowledge

Post by LoneBear » Sun Sep 27, 2009 4:37 pm

Tulan wrote:Does the distribution look something like - across participating members or across institutions such as the Amazon S3 service? Or a mix of both? I personally like the idea of setting up something like Freenet but serving entirely as a data store that members participate in. A bit like the folding at home project in some ways, except more for data store than crunching.
I downloaded Freenet and have been playing with it a bit. One problem I see is that you need a large number of peers for it to work well; might not be viable for just starting out, though may be a good long-term solution. I see they are developing a Freenet filesystem, which would be the way to go.

Post Reply