/[storm]/storm/doc/pegboard/available_overlays--hemppah/peg.rst
ViewVC logotype

Diff of /storm/doc/pegboard/available_overlays--hemppah/peg.rst

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.39 by hemppah, Thu Jul 31 12:56:11 2003 UTC revision 1.40 by hemppah, Fri Aug 1 09:00:50 2003 UTC
# Line 8  PEG available_overlays--hemppah: Availab Line 8  PEG available_overlays--hemppah: Availab
8  :Revision: $Revision$  :Revision: $Revision$
9  :Status:   Current  :Status:   Current
10  :Scope:    Major  :Scope:    Major
11  :Type:     Architecture  :Type:     Implementation
12  :Stakeholders: tjl, benja  :Stakeholders: tjl, benja
 .. Affect-PEGs:  
13    
 The purpose of this PEG document is to give a short overview of existing structured  
 P2P overlays which have an open source implementation.  
14    
15  The reason for creating this PEG document is that the current implementation  The current implementation of GISP seems to have (too) many obvious security
16  of GISP [5]_ seems to have (too) many obvious security exploits. Therefore we need  exploits. Therefore we need to examine other available structured overlays which
17  to examine other available structured overlays which have an open source  have an open source implementation in the hope to find a more mature P2P platform.
18  implementation in hope for finding a more mature P2P platform.  After reviewing available open source overlays we propose that we shall use Tapestry
19    as a P2P platform with Storm/Fenfire.
20    
21  The list of implemented overlays is as of July 2003.  The list of implemented overlays is as of July 2003.
22    
# Line 44  Abstractions Line 42  Abstractions
42          key and a value. This interface implements a simple store          key and a value. This interface implements a simple store
43          and retrieve functionality, where the value is always stored          and retrieve functionality, where the value is always stored
44          at the live overlay node(s) to which the key is mapped by          at the live overlay node(s) to which the key is mapped by
45          the KBR layer. Values can be objects of any type. For ex-ample, the DHT          the KBR layer. Values can be objects of any type. For example, the DHT
46          implemented as part of the DHash inter-face in CFS stores          implemented as part of the DHash interface in CFS stores
47          and retrieves single disk blocks by their content-hashed keys.          and retrieves single disk blocks by their content hashed keys.
48    
49  .. _DOLR:  .. _DOLR:
50    
51      DOLR      DOLR
52          The DOLR abstraction provides a decentralized direc-tory          The DOLR abstraction provides a decentralized directory
53          service. Each object replica (or endpoint) has an          service. Each object replica (or endpoint) has an
54          objectID and may be placed anywhere within the system.          objectID and may be placed anywhere within the system.
55          Applications announce the presence of endpoints by pub-lishing          Applications announce the presence of endpoints by publishing
56          their locations. A client message addressed with          their locations. A client message addressed with
57          a particular objectID will be delivered to a nearby end-point          a particular objectID will be delivered to a nearby endpoint
58          with this name. Note that the underlying distributed          with this name. Note that the underlying distributed
59          directory can be implemented by annotating trees associ-ated          directory can be implemented by annotating trees associated
60          with each objectID; other implementations are pos-sible.          with each objectID; other implementations are possible.
61          One might ask why DOLR is not implemented on          One might ask why DOLR is not implemented on
62          top of a DHT, with data pointers stored as values; this is          top of a DHT, with data pointers stored as values; this is
63          not possible because a DOLR routes messages to the near-est          not possible because a DOLR routes messages to the nearest
64          available endpoint providing a locality property not          available endpoint providing a locality property not
65          supported by DHTs. An integral part of this process is the          supported by DHTs. An integral part of this process is the
66          maintenance of the distributed directory during changes          maintenance of the distributed directory during changes
67          to the underlying nodes or links.          to the underlying nodes or links.
68            
69            
70    Note that DHT and DOLR can be seen as "equal" abstractions, i.e., either
71    one doesn't rely on other: DHT can be implemented without relying DOLR and vice
72    versa.
73    
74  Redundancy      
     What techniques are used for redundancy      
   
 Fault tolerance against hostile nodes  
     What techniques are used against hostile nodes      
75    
76  Activity of development  Activity of development
77      How actively a software is being developed.      How actively a software is being developed.
# Line 89  License Line 88  License
88    
89  Other notes  Other notes
90      Other miscellaneous notes.      Other miscellaneous notes.
91        
92        
93    Additionally, we will mention the community that is developing a software under a
94    "Developer" section. THe two choices are:
95    
96    1) "software engineering community", i.e., a regular free software project.
97    
98    2) "research community, i.e., a research group that consist one or more researcher
99       who developes a software
100    
101  Implemented overlays  Implemented overlays
102  ====================  ====================
103    
104  In this section we list the structured P2P overlays which have an open source  In this section we list the structured P2P overlays which have an open source
105  implementation. Please notice that features described for each implementation  implementation. Please note that the description of each implementation's features
106  are deliberately kept as short as possible. For more depth information about the  is deliberately kept as short as possible. For more in-depth information about the
107  overlays we suggest reading the original publications.  overlays we suggest reading the original publications.
108    
109  Chord [1]_  Chord
110  ----------  -----
111    
112    Homepage
113    http://www.pdos.lcs.mit.edu/chord/
114    
115  Abstraction  Abstraction
116      DHT_/DOLR_      DHT_/DOLR_
# Line 108  Redundancy Line 119  Redundancy
119      Replication, backup links.      Replication, backup links.
120    
121  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
122      (undefined)      Not known.
123        
124    License
125        BSD-like
126        
127    Language
128        C++ (GCC 2.95, not 2.96, although 3.x should work)
129    
130  Activity of development  Activity of development
131      Quite high.      Quite high.
132    
133      According to the Chord [1]_ website:      According to the Chord website:
134    
135          At this point no official release for          At this point no official release for
136          Chord is available, but you are welcome to check out the latest version          Chord is available, but you are welcome to check out the latest version
# Line 123  Activity of development Line 140  Activity of development
140  Developer  Developer
141      MIT (research community)      MIT (research community)
142    
 Language  
     C++ (GCC 2.95, not 2.96, although 3.x should work)  
   
143  Additional requirements  Additional requirements
144      * Self-certifying File System (http://fs.net)      * Self-certifying File System (http://fs.net)
145      * autoconf, automake, etc.      * autoconf, automake, etc.
146      * Berkeley DB 3.X      * Berkeley DB 3.X
147    
 License  
     BSD-like  
   
148  Other notes  Other notes
149      No support for network locality- do not take network latencies into account      No support for network locality- does not take network latencies into account
150      when building neighbor links.      when building neighbor links.
151    
152      Includes a Chord [1]_ simulator.      Includes a Chord simulator.
153        
154        
155            
156    
157  Tapestry [2]_  Tapestry
158  --------------  --------
159    
160    Homepage
161    http://www.cs.berkeley.edu/~ravenben/tapestry/
162    
163  Abstraction  Abstraction
164      DOLR_      DOLR_
# Line 153  Redundancy Line 169  Redundancy
169    
170          Tapestry is highly resilient under dynamic conditions,          Tapestry is highly resilient under dynamic conditions,
171          providing a near optimal success rate for requests          providing a near optimal success rate for requests
172          under high churn rates, and quicly recovering from          under high churn rates, and quickly recovering from
173          massive membership events in under a minute.          massive membership events in under a minute.
174            
175      According to the Tapestry [2]_ website:      According to the Tapestry website:
176    
177          Tapestry offers fault-resilient mechanisms for both object          Tapestry offers fault-resilient mechanisms for both object
178          location and point to point message delivery.  For object location,          location and point to point message delivery.  For object location,
# Line 174  Redundancy Line 190  Redundancy
190    
191            
192  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
193      * PKI is used while creating node identifiers      * PKI is used while creating node identifiers (to prevent `Sybil attacks`_)
194      * MACs are used to maintain integrity of overlay traffic      * MACs are used to maintain integrity of overlay traffic (to maintain integrity
195      * Monitoring system for maintaining neighbor links        of overlay traffic)
196        * Monitoring system for maintaining neighbor links (reduce packet loss/improve
197          message delivery in the overlay)
198      
199    License
200        BSD-like
201    
202    Language
203        Java (Sun JDK 1.3 or a compatible Java Development and Runtime environment).
204        The Java interface libraries for the BerkeleyDB database
205        
206  Activity of development  Activity of development
207      Active.      Active.
208    
209      Tapestry  1.0 (April 2002)      Tapestry  1.0 (April 2002)
210    
211      According to the Tapestry [2]_ website::      According to the Tapestry website::
212    
213          Tapestry Version 1.0 contains the following functionality:          Tapestry Version 1.0 contains the following functionality:
214    
# Line 205  Activity of development Line 230  Activity of development
230            decide when to use backup routes              decide when to use backup routes  
231                
232                
233       According to the Tapestry [2]_ website, Tapestry Version 2.0 contains the       According to the Tapestry website, Tapestry Version 2.0 contains the
234       following new functionality::       following new functionality::
235    
236          * Algorithms for adapting to changing network conditions          * Algorithms for adapting to changing network conditions
# Line 230  Activity of development Line 255  Activity of development
255  Developer  Developer
256      University of Berkeley (research community)      University of Berkeley (research community)
257    
 Language  
     Java (Sun JDK 1.3 or a compatible Java Development and Runtime environment).  
     The Java interface libraries for the BerkeleyDB database  
   
258  Additional requirements  Additional requirements
259      * The Cryptix JCE library (included with the 2.0 release)      * The Cryptix JCE library (included with the 2.0 release)
260      * UNIX make program      * UNIX make program
261      * The Java interface libraries for the BerkeleyDB database      * The Java interface libraries for the BerkeleyDB database
262        (included with the 2.0 release)        (included with the 2.0 release)
       
 License  
     BSD-like  
263    
264  Other notes  Other notes
265      Support for network locality when building neighbor links.      Support for network locality when building neighbor links.
266    
267      Why Oceanstore_ uses Tapestry [2]_?      Why Oceanstore_ uses Tapestry ?
268      See http://www.oceanstore.org/info/whytapestry.html      See http://www.oceanstore.org/info/whytapestry.html
269    
270  Kademlia [3]_  Kademlia
271  -------------  --------
272    
273    Homepage
274    http://kademlia.scs.cs.nyu.edu
275    
276  Abstraction  Abstraction
277      DHT_      DHT_
278        
279  Redundancy  Redundancy
280      No simulation or test results published (not even in the original publication)      No simulation or test results published (not even in the original publication).
281      In *theory*, however, the "free-choice" feature      In *theory*, however, the "free-choice" feature
282      gives peers freedom to adapt different conditions. However, the      gives peers freedom to adapt different conditions. However, the
283      author of SharkyPy says:      author of SharkyPy says:
# Line 277  Redundancy Line 298  Redundancy
298    
299    
300  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
301      Nothing said, expect the "free-choice" feature.      Nothing said, expect the "free-choice" feature.
302        
303    License
304        GPL (Java), "Free for non-commercial use" (C++)
305        
306    Language
307        Java (Sun JDK 1.3 or a compatible Java Development and Runtime environment (?))
308    
309  Activity of development  Activity of development
310      Java development discontinued, C++ version      Java development discontinued, C++ version
# Line 286  Activity of development Line 313  Activity of development
313  Developer  Developer
314      New York University (research community).      New York University (research community).
315    
 Language  
     Java (Sun JDK 1.3 or a compatible Java Development and Runtime environment (?))  
   
 License  
     GPL (Java), "Free for non-commercial use" (C++)  
   
316  Other notes  Other notes
317      The implementation of the Java version is discontinued.      The implementation of the Java version is discontinued.
318    
319  Pastry [4]_  Pastry
320  -----------  ------
321    
322    Homepage
323    http://research.microsoft.com/~antr/Pastry/
324    
325  Abstraction  Abstraction
326      DHT_      DHT_
327        
328  Redundancy  Redundancy
329     Backup links.     Backup links.
330    
# Line 309  Fault tolerance against hostile nodes Line 333  Fault tolerance against hostile nodes
333      this release. Therefore, the  software should only be run in trusted      this release. Therefore, the  software should only be run in trusted
334      environments. Future releases will include security."      environments. Future releases will include security."
335    
336    License
337        BSD-like license (Java), MSR-EULA (C#)
338        
339    Language
340        Java (requires a Java runtime version 1.4), C# (not known)
341    
342  Activity of development  Activity of development
343      Active      Active
344            
# Line 317  Activity of development Line 347  Activity of development
347  Developer  Developer
348      Microsoft Research and Rice University (research community)      Microsoft Research and Rice University (research community)
349    
 Language  
     Java (requires a Java runtime version 1.4), C# (not known)  
   
 License  
     BSD-like license (Java), MSR-EULA (C#)  
   
350  Other notes  Other notes
351      Support for network locality - Pastry [4]_ actively replicates the objects and      Support for network locality - Pastry actively replicates the objects and
352      places them at random locations in the network. Result: When locating      places them at random locations in the network. Result: When locating
353      nearby object it might require the client to route to a distant replica of      nearby object it might require the client to route to a distant replica of
354      the object.      the object.
355            
356      According to the Pastry [4]_ website:      According to the Pastry website:
357    
358          Future releases will address efficiency,          Future releases will address efficiency,
359          security, interoperability and will provide additional          security, interoperability and will provide additional
# Line 337  Other notes Line 361  Other notes
361          hash table implementation, replica management,          hash table implementation, replica management,
362          caching, etc.          caching, etc.
363    
364  GISP [5]_  GISP
365  ---------  ----
366    
367    Homepage
368    http://gisp.jxta.org/
369    
370  Abstraction  Abstraction
371      DHT_/DOLR_      DHT_/DOLR_
372    
373  Redundancy  Redundancy
374      Chord [1]_-like (since GISP uses similar routing tables as Chord)      Chord-like (since GISP uses similar routing tables as Chord)
375    
376  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
377      Based on our own initial experiments: the fault tolerance      Based on our own initial experiments: the fault tolerance
378      is relatively weak - no specific techiques used.      is relatively weak - no specific techiques used.
379    
380    License
381        Sun Project JXTA License Version 1.1
382        
383    Language
384        Java (requires a Java runtime version 1.4)
385    
386  Activity of development  Activity of development
387      Quite active.      Quite active.
388    
389  Developer  Developer
390      Daishi Kato (software engineering community)      Daishi Kato (software engineering community)
391    
 Language  
     Java (requires a Java runtime version 1.4)  
   
 License  
     Sun Project JXTA License Version 1.1  
   
392  Other notes  Other notes
393      Uses 10x more cache as Chord [1]_ for routing table.      Uses 10x more cache as Chord for routing table.
394    
395        Includes a GISP simulator.
396    
397      Includes a GISP [5]_ simulator.  Circle
398    ------
399    
400  Circle [6]_  Homepage
401  -----------  http://thecircle.org.au/
402    
403  Abstraction  Abstraction
404      DHT_      DHT_
# Line 380  Fault tolerance against hostile nodes Line 410  Fault tolerance against hostile nodes
410      According to Info-Anarchy Wiki:      According to Info-Anarchy Wiki:
411    
412          Problems are: The DHT implementation is          Problems are: The DHT implementation is
413          vulnerable to denial of service attacks.              vulnerable to denial of service attacks.
414    
415    License
416        GPL
417    
418    Language
419        Python (version 2.0 or higher, 2.2 preferred, GTK+-2 and PyGTK)
420    
421  Activity of development  Activity of development
422      Active, the current version is 0.35 (30 May 2003)      Active, the current version is 0.35 (30 May 2003)
423    
424  Developer  Developer
425      Paul Harrison (software engineering community)      Paul Harrison (software engineering community)
426    
 Language  
     Python (version 2.0 or higher, 2.2 preferred, GTK+-2 and PyGTK)  
   
 License  
     GPL  
   
427  Other notes  Other notes
428      Uses MD5 hashes for generating IDs.      Uses MD5 hashes for generating IDs.
429            
430    
431  Khashmir [7]_  Khashmir
432  -------------  --------
433    
434    Homepage
435    http://khashmir.sourceforge.net/
436    
437  Abstraction  Abstraction
438      DHT_ (Kademlia [3]_ algorithm)      DHT_ (Kademlia algorithm)
439    
440  Redundancy  Redundancy
441      Not known      Not known.
442    
443  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
444      According to the authors:      According to the authors:
# Line 413  Fault tolerance against hostile nodes Line 446  Fault tolerance against hostile nodes
446          Note that Khashmir currently isn't very          Note that Khashmir currently isn't very
447          attack resistant.          attack resistant.
448    
449    License
450        MIT License
451        
452    Language
453        Python
454        
455  Activity of development  Activity of development
456      Not active      Not active
457            
# Line 421  Activity of development Line 460  Activity of development
460  Developer  Developer
461      Four developers (software engineering community)      Four developers (software engineering community)
462    
 Language  
     Python  
   
 License  
     MIT License  
   
463  Other notes  Other notes
464      (none)      (none)
465    
466  MLDonkey [8]_  MLDonkey
467  -------------  --------
468    
469    Homepage
470    http://www.nongnu.org/mldonkey/
471    
472  Abstraction  Abstraction
473      DHT_ (Kademlia [3]_ algorithm)      DHT_ (Kademlia algorithm)
474    
475      (MLDonkey [8]_ is compatible with Overnet_, and Overnet claims that it does      (MLDonkey is compatible with Overnet_ (non-open source implementation of Kademlia
476      Kademlia [3]_ and multisource downloading)      developed by a company), and Overnet claims that it does Kademlia and multisource
477        downloading).
478    
479  Redundancy  Redundancy
480      Not known (we can imagine that redundancy is relatively high since MLDonkey      Not known (we can imagine that redundancy is relatively high since MLDonkey
481      is widely deplyed)      is widely deplyed)
482    
483  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
484      Not known (we can imagine that fault tolerance is relatively high since MLDonkey      Not "officially" known (we can imagine that fault tolerance is relatively
485      is widely deplyed)      high since MLDonkey is widely deplyed).
486    
487    License
488        GPL
489        
490    Language
491        Objective-Caml (a language that compiler compiling)
492        
493  Activity of development  Activity of development
494      Very active      Very active
495    
# Line 456  Developer Line 499  Developer
499      12 developers (according to Savannah's project page, software engineering      12 developers (according to Savannah's project page, software engineering
500      community)      community)
501    
 Language  
     Objective-Caml (a compiler, not a interpreter)  
   
 License  
     GPL  
   
502  Other notes  Other notes
503      Supported P2P networks include eDonkey, Overnet, Bittorrent,      Supported P2P networks include eDonkey, Overnet, Bittorrent,
504      Gnutella (Bearshare, Limewire,etc),  Gnutella2  (Shareaza),        Gnutella (Bearshare, Limewire,etc),  Gnutella2  (Shareaza),  
# Line 475  Other notes Line 512  Other notes
512      Overnet is not a free specification ,i.e., change control is in the Overnet      Overnet is not a free specification ,i.e., change control is in the Overnet
513      company's hands.      company's hands.
514    
515      According to the `MLDonkey CVS source server`_ (check this_ too), MLDonkey [8]_      According to the `MLDonkey CVS source server`_ (check this_ too), MLDonkey  
516      uses MD4 hashes for Overnet/EDonkey2K IDs::      uses MD4 hashes for Overnet/EDonkey2K IDs::
517            
518          peer: an overnet peer in the following format:          peer: an overnet peer in the following format:
# Line 488  Other notes Line 525  Other notes
525      (http://www.overnet.com/documentation/how.html,      (http://www.overnet.com/documentation/how.html,
526      http://bitzi.com/help/locallinks)      http://bitzi.com/help/locallinks)
527    
528  SharkyPy [9]_  SharkyPy
529  -------------  --------
530    
531    Homepage
532    http://www.heim-d.uni-sb.de/~heikowu/SharkyPy/
533    
534  Abstraction  Abstraction
535      DHT_ (Kademlia [3]_ algorithm)      DHT_ (Kademlia algorithm)
536    
537  Redundancy  Redundancy
538      According to the author:      According to the author:
# Line 504  Redundancy Line 544  Redundancy
544  Fault tolerance against hostile nodes  Fault tolerance against hostile nodes
545      No specific techniques used: "it should work as is"      No specific techniques used: "it should work as is"
546    
547    Language
548        Python
549    
550    License
551        LGPL
552        
553  Activity of development  Activity of development
554      The current version is 0.2b3 (16th February 2003)      The current version is 0.2b3 (16th February 2003)
555    
# Line 568  Activity of development Line 614  Activity of development
614  Developer  Developer
615      Sprachenzentrum der Universität des Saarlandes / Heiko Wundram (research community)      Sprachenzentrum der Universität des Saarlandes / Heiko Wundram (research community)
616    
 Language  
     Python  
   
 License  
     LGPL  
   
617  Other notes  Other notes
618      According to the author of SharkyPy:      According to the author of SharkyPy:
619            
# Line 597  importance: Line 637  importance:
637  - The activity of development  - The activity of development
638  - The implementation language  - The implementation language
639    
640  As a result, we recommend using Tapestry's open source implementation with Storm.  As a result, we recommend using Tapestry's open source implementation for use in
641  For detailed changes and information about using Tapestry with Storm, please  Storm. For a detailed list of changes necessary to use Tapestry in Storm, see
642  see the `storm_with_tapestry--hemppah` PEG document.  the ``storm_with_tapestry--hemppah`` (a pending PEG) PEG document.
643    
644    
 .. [1] http://www.pdos.lcs.mit.edu/chord/  
 .. [2] http://www.cs.berkeley.edu/~ravenben/tapestry/  
 .. [3] http://kademlia.scs.cs.nyu.edu  
 .. [4] http://research.microsoft.com/~antr/Pastry/  
 .. [5] http://gisp.jxta.org/  
 .. [6] http://thecircle.org.au/  
 .. [7] http://khashmir.sourceforge.net/  
 .. [8] http://www.nongnu.org/mldonkey/  
 .. [9] http://www.heim-d.uni-sb.de/~heikowu/SharkyPy/  
645    
646  .. _`Towards a Common API for Structured P2P Overlays`: http://www.cs.berkeley.edu/~ravenben/publications/pdf/apis.pdf  .. _Towards a Common API for Structured P2P Overlays: http://www.cs.berkeley.edu/~ravenben/publications/pdf/apis.pdf
647  .. _`Tapestry: A Resilient Global-scale Overlay for Service Deployment`: http://www.cs.berkeley.edu/~ravenben/publications/pdf/tapestry_jsac.pdf  .. _`Tapestry: A Resilient Global-scale Overlay for Service Deployment`: http://www.cs.berkeley.edu/~ravenben/publications/pdf/tapestry_jsac.pdf
648  .. _Overnet: http://www.overnet.com  .. _Overnet: http://www.overnet.com
649  .. _Oceanstore: http://www.oceanstore.org  .. _Oceanstore: http://www.oceanstore.org
650  .. _`MLDonkey CVS source server`: http://savannah.nongnu.org/cgi-bin/viewcvs/mldonkey/mldonkey/docs/overnet.txt?rev=1.1&content-type=text/vnd.viewcvs-markup  .. _MLDonkey CVS source server: http://savannah.nongnu.org/cgi-bin/viewcvs/mldonkey/mldonkey/docs/overnet.txt?rev=1.1&content-type=text/vnd.viewcvs-markup
651  .. _this: http://savannah.nongnu.org/cgi-bin/viewcvs/mldonkey/mldonkey/src/networks/donkey/donkeyOvernet.ml?rev=1.4&content-type=text/vnd.viewcvs-markup  .. _this: http://savannah.nongnu.org/cgi-bin/viewcvs/mldonkey/mldonkey/src/networks/donkey/donkeyOvernet.ml?rev=1.4&content-type=text/vnd.viewcvs-markup
652  .. _post: http://mail.python.org/pipermail/python-list/2003-February/143876.html  .. _post: http://mail.python.org/pipermail/python-list/2003-February/143876.html
653  .. _`python-list`: http://mail.python.org/mailman/listinfo/python-list  .. _python-list: http://mail.python.org/mailman/listinfo/python-list
654  .. _message: http://mail.python.org/pipermail/python-list/2003-February/148394.html  .. _message: http://mail.python.org/pipermail/python-list/2003-February/148394.html
655    .. _Sybil attacks: http://www.cs.rice.edu/Conferences/IPTPS02/101.pdf

Legend:
Removed from v.1.39  
changed lines
  Added in v.1.40

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26