|
|
|
|
|
|
1 |
1. Approaches |
1. Approaches |
2 |
|
|
3 |
-there are five approaches when performing searches in p2p networks. |
-there are five approaches when performing searches in p2p networks. |
38 |
-SWNs require O(log^2 n) hops to reach arbitrary destinations, assuming (*only and only if* !!!) that |
-SWNs require O(log^2 n) hops to reach arbitrary destinations, assuming (*only and only if* !!!) that |
39 |
links between nodes are constructed in the way that they are uniformly distributed over all distances |
links between nodes are constructed in the way that they are uniformly distributed over all distances |
40 |
in the network (Kleinberg) |
in the network (Kleinberg) |
41 |
|
|
42 |
-Example systems: SWAN, Freenet |
-Example systems: SWAN, Freenet |
43 |
|
|
44 |
|
Req. 1: |
45 |
|
The short-range links must be such that for all nodes n and m, where n != m, n |
46 |
|
has a short-range link to a node l, so that distance(l, m) < distance(n, m). |
47 |
|
|
48 |
|
Notice: The length of the link from node n to node m is distance(n, m) |
49 |
|
|
50 |
|
Req. 2: |
51 |
|
The long-range links must be such that for all Long-range links are nearly |
52 |
|
uniformly distributed over all 'distance scales'. |
53 |
|
|
54 |
|
Notice: There *is* a more formal specification about req. 2, but I won't give it |
55 |
|
right here, because it quite mathematical (and therefore it makes no sense to |
56 |
|
present it here with these characters). |
57 |
|
|
58 |
|
And, if these requirements are met, SWN network can locate any data in O(log^2 |
59 |
|
n) hops (Kleinberg and e.g. simulations in SWAN) |
60 |
|
|
61 |
1.3. Flooding Broadcast Networks (FBN) |
1.3. Flooding Broadcast Networks (FBN) |
62 |
+own resources are not mapped into the network |
+own resources are not mapped into the network |
63 |
+keyword/fuzzy search possible |
+keyword/fuzzy search possible |
105 |
|
|
106 |
** = Please see http://www.darkridge.com/~jpr5/doc/gnutella.html for details |
** = Please see http://www.darkridge.com/~jpr5/doc/gnutella.html for details |
107 |
|
|
108 |
** = From p2p-hackers mailinglist: |
*** = From p2p-hackers mailinglist: |
109 |
- - - |
- - - |
110 |
|
|
111 |
> Btw, how well Alpine can scale (e.g. number of users) ? Do you have any "real- |
> Btw, how well Alpine can scale (e.g. number of users) ? Do you have any "real- |
221 |
-metadata ? |
-metadata ? |
222 |
|
|
223 |
2.1.5. Answers to research problem |
2.1.5. Answers to research problem |
224 |
|
-there are different kinds of uses of DHTs, e.g. DHT actually stores the data, or DHT only stores |
225 |
|
the values, which are used for locating the data |
226 |
|
-current systems mainly uses to latter method |
227 |
|
-both have pros and cons: |
228 |
|
-in the value use, several hostile nodes might say that they hosts the value, but refuse to serve any host |
229 |
|
-in the storage use, hostile node might say that someone must save data block size of 1TB for her computer |
230 |
|
-however, there are methods for preventing these kinds of actions |
231 |
|
-DHTs and SWNs are very robust to failures: e.g. 20% of the nodes simultaneously fail, less than 0.1% of the data retrievals are failed |
232 |
|
-the basic difference between DHTs/SWNs is that in DHT/SWN approach, systems "knows", where the desired data block resides |
233 |
|
-based on data block ID's, nodes gives "hints" (routes more close to ID in the key space) constantly to a query lookup until |
234 |
|
the specific node is found which hosts the block |
235 |
|
-no extra network traffic |
236 |
|
-so in a way, DHT/SWNs are structured entities |
237 |
|
-instead, in other approaches, system doesn't "know" where the data block resides |
238 |
|
-we have to ask from any participating node, if they have the data block |
239 |
|
-very much extra network traffic |
240 |
|
|
241 |
-the most efficient algorithm: O(log n) |
-the most efficient algorithm: O(log n) |
242 |
-DHTs requires O(log n) hops, log n neighbors, except Viceroy (however, robustness suffers) |
-DHTs requires O(log n) hops, log n neighbors, except Viceroy (however, robustness suffers) |
243 |
-SWNs requires O(log^2 n) hops, much less neighbors (constant, is not depedent of n) |
-SWNs requires O(log^2 n) hops, much less neighbors (constant, is not depedent of n) |
247 |
-in this case, FBNs, Hybrids, Social Discovery require much bandwidth, since there is no benefit from the block's ID at all (they have to ask constantly) |
-in this case, FBNs, Hybrids, Social Discovery require much bandwidth, since there is no benefit from the block's ID at all (they have to ask constantly) |
248 |
-SWNs require less memory than DHTs, since there are less connections to other nodes |
-SWNs require less memory than DHTs, since there are less connections to other nodes |
249 |
-SWNs relies less to neighbor nodes than DHTs |
-SWNs relies less to neighbor nodes than DHTs |
250 |
-proposol: most efficient approaches for this research problem are DHTs and SWNs, since they are based on key-value pairs (Storm has a key as block ID) |
-proposol: most efficient approaches for this research problem are DHTs and SWNs, since they are based on key-value pairs (Storm has a key as block ID) |
251 |
|
|
252 |
|
|
253 |
|
|
254 |
|
|
277 |
-In this case (urn-5), there is no benefit from the block's ID at all (DHT, SWN) |
-In this case (urn-5), there is no benefit from the block's ID at all (DHT, SWN) |
278 |
|
|
279 |
2.2.4. Open questions: |
2.2.4. Open questions: |
280 |
-in this case, is it sensible to maintain two different key-value mappings key-value based systems (DHT, SWN) ? |
-in this case, is it sensible to maintain two different key-value mappings key-value based systems (DHT, SWN) ?l |
281 |
-one fore block IDs |
-one fore block IDs |
282 |
-one for urn-5 names, which are associated with block IDs |
-one for urn-5 names, which are associated with block IDs |
283 |
-is this approach too difficult to maintain ? |
-is this approach too difficult to maintain ? |
287 |
-How CAs' data should be saved ? |
-How CAs' data should be saved ? |
288 |
|
|
289 |
2.2.5. Answers to research problem |
2.2.5. Answers to research problem |
290 |
-compared to previous research problem, the "first seen" benefits of DHTs and SWNs in this research problem are smaller |
-compared to previous research problem, the "obvious" benefits of DHTs and SWNs in this research problem are smaller |
291 |
-we don't know which block is the most recent associated with a specfic urn-5 name |
-we don't beforehand know which block is the most recent associated with a specfic urn-5 name |
292 |
-in theory, though, the most efficient algorithm is O(log n), *assuming* that we already know the most recent block's ID (see previos research problem) |
-in theory, though, the most efficient algorithm is O(log n), *assuming* that we already know the most recent block's ID (see previous research problem) |
293 |
-currently, there is no "out-of-the-box" answer to this research problem |
-the most important question is, how we can efficiently associate urn-5 names with the most recent block / to all blocks which are associated with it |
294 |
-the question is, how we can efficiently associate urn-5 names with the most recent block / to all blocks which are associated with it |
-for DHTs and SWNs: |
295 |
For DHTs and SWNs: |
|
296 |
-adaption of Benja's 1st idea: each node maintains a local hash table (urn-5 name -> most recent local block ID) for every urn-5 names |
Please notice: In this approach, DHT doesn't store the actual block, only the values for locating the data from the system |
297 |
which node hosts --> we don't have to check all blocks and their urn-5 associations |
|
298 |
-adaption of Benja's 2nd idea: All urn-5 name mappings are stored as <key, value[ ]>, where the key is urn-5 name's hash and value is a record containing |
Req. 1: |
299 |
|
-each node maintains a local stack based data structure (urn-5 name -> most recent local block ID) for every urn-5 names |
300 |
|
which node hosts. The most recent block is topmost --> we don't have to check all blocks and their urn-5 associations to get the most recent |
301 |
|
Req. 2: |
302 |
|
-all urn-5 name mappings are stored as <key, value[ ]>, where the key is urn-5 name's hash and value is a record containing |
303 |
block ID and timestamp of that block. So, when we store a block in our system first time, we have to create a new key-value: |
block ID and timestamp of that block. So, when we store a block in our system first time, we have to create a new key-value: |
304 |
<hash_of_urn_5_name, [block id, block timestamp]> and route this mapping to node which is "closest" to a hash value. Now when we want to find the most |
<hash_of_urn_5_name, [block id, block timestamp]> and route this mapping to node which is "closest" to a hash value. Now when we want to find the most |
305 |
recent block associated with a specific urn-5 name, we do: |
recent block associated with a specific urn-5 name, we do: |
306 |
|
|
307 |
1. locally compute a hash for given urn-5 string |
1. locally compute a hash for given urn-5 string |
308 |
2. route the query to a node which hosts the given hash of urn-5 ("closest" node) |
2. route the query to a node which hosts the given hash of urn-5 ("closest" node) |
309 |
3. get most recent block's ID using the idea from previuos idea |
3. get most recent block's ID using the idea from previuos idea |
310 |
|
4. use ID to get the specific block using normal DHT/SWN operation |
311 |
|
|
312 |
In this approach, we don't have to perform additional searching and sorting of mappings. And of course, we know that for given urn-5, only one node |
In this approach, we don't have to perform additional searching and sorting of mappings. And of course, we know that for given urn-5, only one node |
313 |
hosts *all* the block information ("block history") for the urn-5, since mappings are mapped to a single node, closest to urn-5 hash value. |
hosts *all* the block information ("block history") for the urn-5, since mappings are mapped to a single node, closest to urn-5 hash value. |
314 |
Again, this should work fine under existing DHTs (and SWTs ?). |
Again, this should work fine under existing DHTs (and SWTs ?). |
315 |
|
|
316 |
|
Some simple analysis: |
317 |
|
-there are more key-value pairs in the system for additional urn-5 --> block associations |
318 |
|
-however, I don't think this is an issue, since data's size is small |
319 |
|
-efficiency: find node which hosts urn-5 names + find node which hosts blocks associated with urn-5 name: logn + logn = 2logn (logarithmical) |
320 |
|
|
321 |
|
-for FBS and others: |
322 |
|
-there is no very efficient (simple) methods for finding urn-5 name associated with the most recent block |
323 |
|
-one simple proposal is that when we visit to each node (first idea above), get only the most recent one and compare them (or greedy approach: dismiss currently |
324 |
|
most recent block as we visit to nodes, if newer block have been found) |
325 |
|
|
326 |
2.3. "Searching for Storm blocks associated with specific urn-5 name, where |
2.3. "Searching for Storm blocks associated with specific urn-5 name, where |
327 |
specific date has been defined (or date range), and where Storm block |
specific date has been defined (or date range), and where Storm block |
351 |
tree based structure for all blocks for specific urn-5 name) ? |
tree based structure for all blocks for specific urn-5 name) ? |
352 |
-How CAs' data should be saved ? |
-How CAs' data should be saved ? |
353 |
|
|
354 |
|
2.3.5. Answers to research problem |
355 |
|
This is quite similar to previous research problem's answer. There are slight differences, though. |
356 |
|
|
357 |
|
-for DHTs and SWNs: |
358 |
|
|
359 |
|
Please notice: In this approach, DHT doesn't store the actual block, only the values for locating the data from the system |
360 |
|
|
361 |
|
Req. 1: |
362 |
|
-each node maintains a local stack based data structure (urn-5 name -> most recent local block ID) for every urn-5 names |
363 |
|
which node hosts. The most recent block is topmost --> we don't have to check all blocks and their urn-5 associations to get the most recent |
364 |
|
Req. 2: |
365 |
|
-all urn-5 name mappings are stored as <key, value[ ]>, where the key is urn-5 name's hash and value is a record containing |
366 |
|
block ID and timestamp of that block. So, when we store a block in our system first time, we have to create a new key-value: |
367 |
|
<hash_of_urn_5_name, [block id, block timestamp]> and route this mapping to node which is "closest" to a hash value. Now when we want to find the most |
368 |
|
recent block associated with a specific urn-5 name, we do: |
369 |
|
|
370 |
|
1. locally compute a hash for given urn-5 string |
371 |
|
2. route the query to a node which hosts the given hash of urn-5 ("closest" node) |
372 |
|
3. get the specific block(s) data (block ID) from the stack which matches to given date&time properties (we compare to block header data) |
373 |
|
4. use IDs to get the specific blocks using normal DHT/SWN operation |
374 |
|
|
375 |
|
Some simple analysis: |
376 |
|
-there are more key-value pairs in the system for additional urn-5 --> block associations |
377 |
|
-however, I don't think this is an issue, since data's size is small |
378 |
|
-efficiency: find node which hosts urn-5 names + find node which hosts blocks associated with urn-5 name: logn + logn = 2logn (logarithmical) |
379 |
|
|
380 |
|
-for FBS and others: |
381 |
|
-there is no very efficient (simple) methods for finding urn-5 name associated with the most recent block |
382 |
|
-one simple proposal is that when we be that we visit to each node (first idea above), get all blocks which matches to given properties |
383 |
|
|
384 |
|
|
385 |
2.4. "How does search engine should work?" |
2.4. "How does search engine should work?" |