/[gzz]/gzz/Documentation/misc/hemppah-progradu/research_problems
ViewVC logotype

Diff of /gzz/Documentation/misc/hemppah-progradu/research_problems

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.4 by hemppah, Tue Dec 17 08:27:50 2002 UTC revision 1.5 by hemppah, Tue Dec 17 10:36:21 2002 UTC
# Line 47  in the network (Kleinberg) Line 47  in the network (Kleinberg)
47  +keyword/fuzzy search possible  +keyword/fuzzy search possible
48  -not scalable  -not scalable
49  -huge network traffic  -huge network traffic
50  -not fast routing (in fact, there is no upper limit, because there are multiple simultaneous  -not fast routing
 breadh-first searches in the network)  
51  -no guarantee that all data will be located  -no guarantee that all data will be located
52    
53  -Example systems: Gnutella, Fastrack family (Kazaa, Morpheus), JXTA Search, Gnutella2  -Example systems: Gnutella, Fastrack family (Kazaa, Morpheus), JXTA Search, Gnutella2
# Line 82  Tapestry:      O(log^2 n)      O(log n)        O(log n) Line 81  Tapestry:      O(log^2 n)      O(log n)        O(log n)
81  Kademlia:       O(log n)*       O(log n)        O(log n)  Kademlia:       O(log n)*       O(log n)        O(log n)
82  Viceroy:        O(log n)        O(1)            O(log n)  Viceroy:        O(log n)        O(1)            O(log n)
83  Small Worlds:   O(1)            O(1)            O(log^2 n)  Small Worlds:   O(1)            O(1)            O(log^2 n)
84  Flooding:       N/A             O(1)            "high"  Flooding:       N/A**           N/A**           N/A**
85  Hybrid:         N/A             N/A             N/A  Hybrid:         N/A             N/A             N/A
86  Social:         N/A             O(1)            N/A  Social:         N/A***          N/A***          N/A***
87    
88  * = In Kademlia, there is no action required when nodes leaves the system  * = In Kademlia, there is no action required when nodes leaves the system
89    
90    ** = Please see http://www.darkridge.com/~jpr5/doc/gnutella.html for details
91    
92    ** = From p2p-hackers mailinglist:
93    - - -
94    
95    > Btw, how well Alpine can scale (e.g. number of users) ? Do you have any "real-
96    > life" experiences from Alpine (how social-connection paradigm really works
97    > etc.) ? What about search performance in Alpine (any big O's) ? Security
98    > (PKIs) ?
99    
100    Scalability is the big question, as it is hard to guage real world
101    scalability without an actual real world network :-)
102    
103    I have done some preliminary scalability testing on the OSDL systems
104    ( http://www.osdl.org/ ) using a pair of 4way SMP systems on a gigabit
105    network link.  I was able to establish and communicate with over 4 million
106    concurrent DTCP/Alpine connections between them, using about a gigabyte of
107    RAM.
108    
109    On lower end hardware and bandwidth (5 - 10M of memory, DSL/cable) peers
110    groups would be much smaller, 10,000 to 25,000 concurrent.
111    
112    The main feature of alpine that allows it to scale with less effort is the
113    fact that all communication is direct, using a lightweight UDP based
114    transport.  The number of connections is limited only by the memory and
115    bandwidth you have available to use.
116    
117    Search performance is also difficult to guage because of the social
118    discovery mechanism employed.  Each peer is continually tuning peer groups
119    to increase the use of high quality peers with similar interests and
120    removing peers that do not contribute or share interests.
121    
122    When you first join the network your query effectiveness is going to be
123    low.  As you use the network, query effectiveness increases given the
124    feedback received from previous queries and how they affect the composition
125    of the peer groups you use for resource discovery.
126    
127    This makes it nice for people who share interests in unpopular or obscure
128    resources; they can eventually obtain a fast, effective peer group for
129    finding these resources.  This is in direct contrast to most other search
130    mechanisms where obscure or unpopular resources are always more difficult
131    to locate.
132    
133    I am focusing on usability for the next and future devel snapshots of
134    alpine, so hopefully some real world use on a wider scale will allow
135    me to answet your questions which much more detail in the future.
136    Right now all I can provide you is rough guesstimates and intuitions..
137    
138    Last, regarding security, I would like to integrate PGP/GPG style
139    assymetric cyphers and digital signatures, however this is a low
140    priority.  I am also considering an implementation of peer frogs
141    ( http://cubicmetercrystal.com/peerfrog/ ) for avoiding man-in-the-middle
142    attacks in large decentralized peer networks (although this is geared
143    more for large wireless peer networks)
144    
145    - - -
146    
147  Insert/Delete:  Insert/Delete:
148  Number of messages when a node joins or leaves the network.  Number of messages when a node joins or leaves the network.
149    
# Line 137  c) FBN Line 193  c) FBN
193  -obviously a lot of extra traffic arises in the network  -obviously a lot of extra traffic arises in the network
194  -approach specific pseudo thinking: "ask repeating from each node if it has block, whose ID value is X"  -approach specific pseudo thinking: "ask repeating from each node if it has block, whose ID value is X"
195    
 d) HS  
 (-text missing)  
   
 e) SDS  
 (-text missing)  
196    
197  2.1.4. Open questions  2.1.4. Open questions
198  (-This case is quite straight forward, since it resembles very much ordinary "locate file and get it")  (-This case is quite straight forward, since it resembles very much ordinary "locate file and get it")
199  -Really, how much better are the second generation FBN systems (Gnutella2) compared to the first generation FBN systems (Gnutella)  -really, how much better are the second generation FBN systems (Gnutella2) compared to the first generation FBN systems (Gnutella)
200          -efficiency ?          -efficiency ?
201          -netowork traffic ?          -netowork traffic ?
202          -will the data will be located, if it exists in the network ?          -will the data will be located, if it exists in the network ?
203          -scalability          -scalability
204    -when searhing, do we have to know the exact hash of block ID or is there other alternatives ?
205            -metadata ?
206            
207    2.1.5. Answers to research problem
208    -the most efficient algorithm: O(log n)
209            -DHTs requires O(log n) hops, log n neighbors, except Viceroy (however, robustness suffers)
210            -SWNs requires O(log^2 n) hops, much less neighbors (constant, is not depedent of n)
211    -in DHTs and SWNs, it is possible to locate data in constant time, BUT it requires knowing all n neighbors!!!
212    -in basic scenario, DHTs and SWTs assume that we have to know value's key beforehand
213    -in this case, DHTs and SWNs require little bandwidth, because "network knows where the block resides"
214    -in this case, FBNs, Hybrids, Social Discovery require much bandwidth, since there is no benefit from the block's ID at all (they have to ask constantly)
215    -SWNs require less memory than DHTs, since there are less connections to other nodes
216    -SWNs relies less to neighbor nodes than DHTs
217    -proposol: most efficient approaches for this research problem are DHTs and SWNs, since they are based on key-value pairs (Storm has a key as block ID)
218    
219    
220    
221  2.2. "Searching for most recent Storm block associated with specific urn-5 name, where  2.2. "Searching for most recent Storm block associated with specific urn-5 name, where
222       the block has been signed with a given key"       the block has been signed with a given key"
# Line 180  return the most recent block, if the sig Line 247  return the most recent block, if the sig
247          -one fore block IDs          -one fore block IDs
248          -one for urn-5 names, which are associated with block IDs          -one for urn-5 names, which are associated with block IDs
249          -is this approach too difficult to maintain ?          -is this approach too difficult to maintain ?
250            
           
251  -is there possibility, in the specific urn-5 name, to maintain information about most recent block's ID for better search performance (or moreover,  -is there possibility, in the specific urn-5 name, to maintain information about most recent block's ID for better search performance (or moreover,
252  tree based structure for all blocks for specific urn-5 name) ?  tree/list based structure for all blocks for specific urn-5 name) ?
253  -How CAs' data should be saved ?  -How CAs' data should be saved ?
254    
255    2.2.5. Answers to research problem
256    -compared to previous research problem, the benefits of DHTs and SWNs in this research problem are smaller
257            -we don't know which block is the most recent associated with a specfic urn-5 name
258    -in theory, though, the most efficient algorithm is O(log n), *assuming* that we already know the most recent block's ID (see previos research problem)
259    -currently, there is no "out-of-the-box" answer to this research problem
260    -the question is, how we can efficiently associate urn-5 names with the most recent block / to all blocks which are associated with it
261            For DHTs and SWNs:
262            -adaption of Benja's 1st idea: each node maintains a local hash table (urn-5 name -> most recent local block ID) for every urn-5 names
263            which node hosts --> we don't have to check all blocks and their urn-5 associations
264            -adaption of Benja's 2nd idea: All urn-5 name mappings are stored as <key, value[ ]>, where the key is urn-5 name's hash and value is a record containing
265            block ID and timestamp of that block. So, when we store a block in our system first time, we have to create a new key-value:
266            <hash_of_urn_5_name, [block id, block timestamp]> and route this mapping to node which is "closest" to a hash value. Now when we want to find the most
267            recent block associated with a specific urn-5 name, we do:      
268                    1. locally compute a hash for given urn-5 string
269                    2. route the query to a node which hosts the given hash of urn-5 ("closest" node)
270                    3. get most recent block's ID using the idea from previuos idea
271                    
272            In this approach, we don't have to perform additional searching and sorting of mappings. And of course, we know that for given urn-5, only one node
273            hosts *all* the block information ("block history") for the urn-5, since mappings are mapped to a single node, closest to urn-5 hash value.
274            Again, this should work fine under existing DHTs (and SWTs ?).  
275    
276  2.3. "Searching for Storm blocks associated with specific urn-5 name, where  2.3. "Searching for Storm blocks associated with specific urn-5 name, where
277       specific date has been defined (or date range), and where Storm block       specific date has been defined (or date range), and where Storm block

Legend:
Removed from v.1.4  
changed lines
  Added in v.1.5

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26