47 |
+keyword/fuzzy search possible |
+keyword/fuzzy search possible |
48 |
-not scalable |
-not scalable |
49 |
-huge network traffic |
-huge network traffic |
50 |
-not fast routing (in fact, there is no upper limit, because there are multiple simultaneous |
-not fast routing |
|
breadh-first searches in the network) |
|
51 |
-no guarantee that all data will be located |
-no guarantee that all data will be located |
52 |
|
|
53 |
-Example systems: Gnutella, Fastrack family (Kazaa, Morpheus), JXTA Search, Gnutella2 |
-Example systems: Gnutella, Fastrack family (Kazaa, Morpheus), JXTA Search, Gnutella2 |
81 |
Kademlia: O(log n)* O(log n) O(log n) |
Kademlia: O(log n)* O(log n) O(log n) |
82 |
Viceroy: O(log n) O(1) O(log n) |
Viceroy: O(log n) O(1) O(log n) |
83 |
Small Worlds: O(1) O(1) O(log^2 n) |
Small Worlds: O(1) O(1) O(log^2 n) |
84 |
Flooding: N/A O(1) "high" |
Flooding: N/A** N/A** N/A** |
85 |
Hybrid: N/A N/A N/A |
Hybrid: N/A N/A N/A |
86 |
Social: N/A O(1) N/A |
Social: N/A*** N/A*** N/A*** |
87 |
|
|
88 |
* = In Kademlia, there is no action required when nodes leaves the system |
* = In Kademlia, there is no action required when nodes leaves the system |
89 |
|
|
90 |
|
** = Please see http://www.darkridge.com/~jpr5/doc/gnutella.html for details |
91 |
|
|
92 |
|
** = From p2p-hackers mailinglist: |
93 |
|
- - - |
94 |
|
|
95 |
|
> Btw, how well Alpine can scale (e.g. number of users) ? Do you have any "real- |
96 |
|
> life" experiences from Alpine (how social-connection paradigm really works |
97 |
|
> etc.) ? What about search performance in Alpine (any big O's) ? Security |
98 |
|
> (PKIs) ? |
99 |
|
|
100 |
|
Scalability is the big question, as it is hard to guage real world |
101 |
|
scalability without an actual real world network :-) |
102 |
|
|
103 |
|
I have done some preliminary scalability testing on the OSDL systems |
104 |
|
( http://www.osdl.org/ ) using a pair of 4way SMP systems on a gigabit |
105 |
|
network link. I was able to establish and communicate with over 4 million |
106 |
|
concurrent DTCP/Alpine connections between them, using about a gigabyte of |
107 |
|
RAM. |
108 |
|
|
109 |
|
On lower end hardware and bandwidth (5 - 10M of memory, DSL/cable) peers |
110 |
|
groups would be much smaller, 10,000 to 25,000 concurrent. |
111 |
|
|
112 |
|
The main feature of alpine that allows it to scale with less effort is the |
113 |
|
fact that all communication is direct, using a lightweight UDP based |
114 |
|
transport. The number of connections is limited only by the memory and |
115 |
|
bandwidth you have available to use. |
116 |
|
|
117 |
|
Search performance is also difficult to guage because of the social |
118 |
|
discovery mechanism employed. Each peer is continually tuning peer groups |
119 |
|
to increase the use of high quality peers with similar interests and |
120 |
|
removing peers that do not contribute or share interests. |
121 |
|
|
122 |
|
When you first join the network your query effectiveness is going to be |
123 |
|
low. As you use the network, query effectiveness increases given the |
124 |
|
feedback received from previous queries and how they affect the composition |
125 |
|
of the peer groups you use for resource discovery. |
126 |
|
|
127 |
|
This makes it nice for people who share interests in unpopular or obscure |
128 |
|
resources; they can eventually obtain a fast, effective peer group for |
129 |
|
finding these resources. This is in direct contrast to most other search |
130 |
|
mechanisms where obscure or unpopular resources are always more difficult |
131 |
|
to locate. |
132 |
|
|
133 |
|
I am focusing on usability for the next and future devel snapshots of |
134 |
|
alpine, so hopefully some real world use on a wider scale will allow |
135 |
|
me to answet your questions which much more detail in the future. |
136 |
|
Right now all I can provide you is rough guesstimates and intuitions.. |
137 |
|
|
138 |
|
Last, regarding security, I would like to integrate PGP/GPG style |
139 |
|
assymetric cyphers and digital signatures, however this is a low |
140 |
|
priority. I am also considering an implementation of peer frogs |
141 |
|
( http://cubicmetercrystal.com/peerfrog/ ) for avoiding man-in-the-middle |
142 |
|
attacks in large decentralized peer networks (although this is geared |
143 |
|
more for large wireless peer networks) |
144 |
|
|
145 |
|
- - - |
146 |
|
|
147 |
Insert/Delete: |
Insert/Delete: |
148 |
Number of messages when a node joins or leaves the network. |
Number of messages when a node joins or leaves the network. |
149 |
|
|
193 |
-obviously a lot of extra traffic arises in the network |
-obviously a lot of extra traffic arises in the network |
194 |
-approach specific pseudo thinking: "ask repeating from each node if it has block, whose ID value is X" |
-approach specific pseudo thinking: "ask repeating from each node if it has block, whose ID value is X" |
195 |
|
|
|
d) HS |
|
|
(-text missing) |
|
|
|
|
|
e) SDS |
|
|
(-text missing) |
|
196 |
|
|
197 |
2.1.4. Open questions |
2.1.4. Open questions |
198 |
(-This case is quite straight forward, since it resembles very much ordinary "locate file and get it") |
(-This case is quite straight forward, since it resembles very much ordinary "locate file and get it") |
199 |
-Really, how much better are the second generation FBN systems (Gnutella2) compared to the first generation FBN systems (Gnutella) |
-really, how much better are the second generation FBN systems (Gnutella2) compared to the first generation FBN systems (Gnutella) |
200 |
-efficiency ? |
-efficiency ? |
201 |
-netowork traffic ? |
-netowork traffic ? |
202 |
-will the data will be located, if it exists in the network ? |
-will the data will be located, if it exists in the network ? |
203 |
-scalability |
-scalability |
204 |
|
-when searhing, do we have to know the exact hash of block ID or is there other alternatives ? |
205 |
|
-metadata ? |
206 |
|
|
207 |
|
2.1.5. Answers to research problem |
208 |
|
-the most efficient algorithm: O(log n) |
209 |
|
-DHTs requires O(log n) hops, log n neighbors, except Viceroy (however, robustness suffers) |
210 |
|
-SWNs requires O(log^2 n) hops, much less neighbors (constant, is not depedent of n) |
211 |
|
-in DHTs and SWNs, it is possible to locate data in constant time, BUT it requires knowing all n neighbors!!! |
212 |
|
-in basic scenario, DHTs and SWTs assume that we have to know value's key beforehand |
213 |
|
-in this case, DHTs and SWNs require little bandwidth, because "network knows where the block resides" |
214 |
|
-in this case, FBNs, Hybrids, Social Discovery require much bandwidth, since there is no benefit from the block's ID at all (they have to ask constantly) |
215 |
|
-SWNs require less memory than DHTs, since there are less connections to other nodes |
216 |
|
-SWNs relies less to neighbor nodes than DHTs |
217 |
|
-proposol: most efficient approaches for this research problem are DHTs and SWNs, since they are based on key-value pairs (Storm has a key as block ID) |
218 |
|
|
219 |
|
|
220 |
|
|
221 |
2.2. "Searching for most recent Storm block associated with specific urn-5 name, where |
2.2. "Searching for most recent Storm block associated with specific urn-5 name, where |
222 |
the block has been signed with a given key" |
the block has been signed with a given key" |
247 |
-one fore block IDs |
-one fore block IDs |
248 |
-one for urn-5 names, which are associated with block IDs |
-one for urn-5 names, which are associated with block IDs |
249 |
-is this approach too difficult to maintain ? |
-is this approach too difficult to maintain ? |
250 |
|
|
|
|
|
251 |
-is there possibility, in the specific urn-5 name, to maintain information about most recent block's ID for better search performance (or moreover, |
-is there possibility, in the specific urn-5 name, to maintain information about most recent block's ID for better search performance (or moreover, |
252 |
tree based structure for all blocks for specific urn-5 name) ? |
tree/list based structure for all blocks for specific urn-5 name) ? |
253 |
-How CAs' data should be saved ? |
-How CAs' data should be saved ? |
254 |
|
|
255 |
|
2.2.5. Answers to research problem |
256 |
|
-compared to previous research problem, the benefits of DHTs and SWNs in this research problem are smaller |
257 |
|
-we don't know which block is the most recent associated with a specfic urn-5 name |
258 |
|
-in theory, though, the most efficient algorithm is O(log n), *assuming* that we already know the most recent block's ID (see previos research problem) |
259 |
|
-currently, there is no "out-of-the-box" answer to this research problem |
260 |
|
-the question is, how we can efficiently associate urn-5 names with the most recent block / to all blocks which are associated with it |
261 |
|
For DHTs and SWNs: |
262 |
|
-adaption of Benja's 1st idea: each node maintains a local hash table (urn-5 name -> most recent local block ID) for every urn-5 names |
263 |
|
which node hosts --> we don't have to check all blocks and their urn-5 associations |
264 |
|
-adaption of Benja's 2nd idea: All urn-5 name mappings are stored as <key, value[ ]>, where the key is urn-5 name's hash and value is a record containing |
265 |
|
block ID and timestamp of that block. So, when we store a block in our system first time, we have to create a new key-value: |
266 |
|
<hash_of_urn_5_name, [block id, block timestamp]> and route this mapping to node which is "closest" to a hash value. Now when we want to find the most |
267 |
|
recent block associated with a specific urn-5 name, we do: |
268 |
|
1. locally compute a hash for given urn-5 string |
269 |
|
2. route the query to a node which hosts the given hash of urn-5 ("closest" node) |
270 |
|
3. get most recent block's ID using the idea from previuos idea |
271 |
|
|
272 |
|
In this approach, we don't have to perform additional searching and sorting of mappings. And of course, we know that for given urn-5, only one node |
273 |
|
hosts *all* the block information ("block history") for the urn-5, since mappings are mapped to a single node, closest to urn-5 hash value. |
274 |
|
Again, this should work fine under existing DHTs (and SWTs ?). |
275 |
|
|
276 |
2.3. "Searching for Storm blocks associated with specific urn-5 name, where |
2.3. "Searching for Storm blocks associated with specific urn-5 name, where |
277 |
specific date has been defined (or date range), and where Storm block |
specific date has been defined (or date range), and where Storm block |