/[gzz]/manuscripts/storm/article.rst
ViewVC logotype

Diff of /manuscripts/storm/article.rst

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.60 by benja, Fri Jan 31 06:57:16 2003 UTC revision 1.61 by hemppah, Fri Jan 31 10:58:56 2003 UTC
# Line 28  In today's computing world, documents mo Line 28  In today's computing world, documents mo
28  computers, being sent as e-mail attachments, carried around on disks,  computers, being sent as e-mail attachments, carried around on disks,
29  published on the web, moved between desktop and laptop systems,  published on the web, moved between desktop and laptop systems,
30  downloaded for off-line reading or copied between computers in a LAN.  downloaded for off-line reading or copied between computers in a LAN.
31  Often, the same document will be independently modified  Often, the same document is independently modified
32  on two unconnected systems. In this paper, we address two issues  on two unconnected, separete systems. We address two issues
33  raised by this *data mobility*: Dangling links, and keeping track  raised by this *data mobility*: Dangling links, and keeping track
34  of alternative versions. Resolvable location-independent identifiers  of alternative versions. Resolvable location-independent identifiers
35  make these issues much easier to deal with, since data  make these issues much easier to deal with, since data
# Line 43  byte sequences identified by cryptograph Line 43  byte sequences identified by cryptograph
43  private data and documents published on the Internet by  private data and documents published on the Internet by
44  using the same identifiers for both.  using the same identifiers for both.
45  Storm has been partially implemented as a part of the Gzz project [ref],  Storm has been partially implemented as a part of the Gzz project [ref],
46  which uses it exclusively for all disk storage. On top of Storm,  which uses Storm exclusively for all disk storage. On top of Storm,
47  we have built a system for storing mutable, versioned data  we have built a system for storing mutable, versioned data
48  and an implementation of Xanalogical storage [ref].  and an implementation of Xanalogical storage [ref].
49    
# Line 213  and keeping track of alternative version Line 213  and keeping track of alternative version
213  3. Block storage  3. Block storage
214  ================  ================
215    
216    [Do we need a figure, which shows the overall structure of block storage
217    with pointers and diffs ? -Hermanni]
218    
219  In our system, Storm (for *storage module*), all data is stored  In our system, Storm (for *storage module*), all data is stored
220  as *blocks*, byte sequences identified by a SHA-1 cryptographic content-hash  as *blocks*, byte sequences identified by a SHA-1 cryptographic content-hash
221  [ref SHA-1 and our ht'02 paper]. Blocks often have a similar granularity  [ref SHA-1 and our ht'02 paper]. Blocks often have a similar granularity
# Line 221  byte sequence would change the hash (and Line 224  byte sequence would change the hash (and
224  Mutable data structures are built on top of the immutable blocks  Mutable data structures are built on top of the immutable blocks
225  (see Section 6).  (see Section 6).
226    
227  hemppah: Or should these lines be inserted to some other section and tell more about these  [Or should these lines be inserted to some other section and tell more about these
228  systems, e.g. 5.2 ?  systems, e.g. 5.2 ? -Hermanni]
229    
230  CFS [ref], which is built upon Chord routing layer[ref], store data as blocks.  CFS [ref], which is built upon Chord routing layer[ref], store data as blocks.
231  However, CFS *splits* files into several miniblocks and spreads blocks over the  However, CFS *splits* files into several miniblocks and spreads blocks over the
# Line 230  available CFS servers. Freenet [ref] and Line 233  available CFS servers. Freenet [ref] and
233  files into blocks, since they store data as whole files. All previously mentioned  files into blocks, since they store data as whole files. All previously mentioned
234  systems lack of the immutable property which is used in Storm blocks.  systems lack of the immutable property which is used in Storm blocks.
235    
236  Immutable blocks has several benefits...  Immutable blocks has several benefits over existing systems...
237    
238  Block storage makes it easy to replicate data between systems.  1) Storm's block storage makes it easy to replicate data between systems.
239  Different versions of the same document can easily coexist at this level,  Different versions of the same document can easily coexist at this level,
240  stored in different blocks. To replicate all data from computer A  stored in different blocks.
241    [Previous sentence doesn't parse to me (what level ?) :( -Hermanni]
242    To replicate all data from computer A
243  on computer B, it suffices to copy all blocks from A to B that B  on computer B, it suffices to copy all blocks from A to B that B
244  does not already store.  does not already store.
245    [Example of Lotus Notes' replication conficts ? -Hermanni]
246    
247  Storm blocks are MIME messages [ref MIME], i.e., objects with  2) Storm blocks are MIME messages [ref MIME], i.e., objects with
248  a header and body as used in Internet mail or HTTP.  a header and body as used in Internet mail or HTTP.
249  This allows them to carry any metadata that can be carried  This allows them to carry any metadata that can be carried
250  in a MIME header, most importantly a content type.  in a MIME header, most importantly a content type.
# Line 250  the following interface:: Line 256  the following interface::
256      get(id) -> block      get(id) -> block
257      add(block)      add(block)
258      delete(block)      delete(block)
259        
260    [analogy to regular Hash Table/DHT ? -Hermanni]
261    
262  Implementations may store blocks in RAM, in individual files,  3) Implementations may store blocks in RAM, in individual files,
263  in a Zip archive, in a database or through other means.  in a Zip archive, in a database or through other means.
264  We have implemented the first three (using hexadecimal  We have implemented the first three (using hexadecimal
265  representations of the block ids for file names).  representations of the block ids for file names).
266    
267  Storing all data in Storm blocks provides *reliability*:  4) Storing all data in Storm blocks provides *reliability*:
268  When saving a document, an application will only *add* blocks,  When saving a document, an application will only *add* blocks,
269  never overwrite existing data. When a bug causes an application  never overwrite existing data. When a bug causes an application
270  to write malformed data, only the changes from one session  to write malformed data, only the changes from one session
271  will be lost; the previous version of the data will still  will be lost; the previous version of the data will still
272  be accessible. This makes Storm well suited as a basis  be accessible. (Footnote: This makes Storm well suited as a basis
273  for implementing experimental projects (such as ours).  for implementing experimental projects (such as ours).)
274    
275  When used in a network environment, Storm ids do not provide  5) When used in a network environment, Storm ids do not provide
276  a hint as to where in the network the matching block can be found.  a hint as to where in the network the matching block can be found.
277  However, current peer-to-peer systems could be used to  However, current peer-to-peer systems could be used to
278  find blocks in a distributed fashion; for example, Freenet [ref],  find blocks efficiently in a distributed fashion; for example,
279  a few recent Gnutella clients [e.g. ref: shareaza] , Overnet/eDonkey2000 [ref]  Freenet [ref], a few recent Gnutella clients [e.g. ref: shareaza],
280  also use SHA-1-based identifiers [e.g. ref: magnet uri].  Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers
281  However, we have not put a network implementation into regular use  [e.g. ref: magnet uri].
282    (Footnote:However, we have not put a network implementation into regular use
283  yet and thus can only describe our design, not report on  yet and thus can only describe our design, not report on
284  implementation experience.  implementation experience.)
285  We discuss peer-to-peer implementations in Section 7, below.  We discuss peer-to-peer implementations in Section 7, below.
286    
287  The immutability of blocks should make caching trivial, since it is  6) The immutability of blocks should make caching trivial, since it is
288  never necessary to check for new versions of blocks.  never necessary to check for new versions of blocks.
289  Since the same namespace is used for local data and data  Since the same namespace [mention urn-5 ? -Hermanni] is used for local data and data
290  retrieved from the network, online documents that have been  retrieved from the network, online documents that have been
291  permanently downloaded to the local harddisk can also be found  permanently downloaded to the local harddisk can also be found
292  by the caching mechanism. This is convenient for offline browsing,  by the caching mechanism. This is convenient for offline browsing,
# Line 285  for example in mobile environments: User Line 294  for example in mobile environments: User
294  while they are online, store them locally, and be sure that  while they are online, store them locally, and be sure that
295  their software will be able to access them as if downloaded  their software will be able to access them as if downloaded
296  from the net, without broken links.  from the net, without broken links.
297    [Previous sentence doesn't parse to me: more simple :( -Hermanni]
298    
299  Given a peer-to-peer distribution mechanism, it would be possible  Given a peer-to-peer distribution mechanism, it would be possible
300  to retrieve blocks from any peer online that has a copy  to retrieve blocks from any peer online that has a copy
301  in its cache or permanent storage. This is similar to the Squirrel  in its cache or permanent storage. This is similar to the Squirrel
302  web cache [ref], but does not require trust between the peers,  web cache [ref] [more refs? -Hermanni], but does not require trust
303  since it is possible to check the blocks' cryptographic hashes.  between the peers, since it is possible to check the blocks' integrity by using
304  Since much-requested blocks would be cached on many systems,  cryptographic hashes. Since much-requested blocks would be
305  such a network could deal with XXX much more easily.  cached on many systems, such a network could deal with XXX
306  On the other hand, there are privacy concerns with exposing  much more easily. On the other hand, there are privacy
307  one's browser cache to the outside world.  concerns with exposing one's browser cache to the outside world.
   
308    
309    
310    [Merge this paragraph with 5) ? -Hermanni]
311  That all data is stored in blocks means that links to it  That all data is stored in blocks means that links to it
312  are completely independent of location; when data is moved  are completely independent of location; when data is moved
313  between servers, references to it do not break. (Of course, this  between servers, references to it do not break. (Footnote: Of course,
314  requires that the blocks can be found no matter what server  this requires that the blocks can be found no matter what server
315  they are on. Again, see Section 7.)  they are on. Again, see Section 7.)
316    
317    [Is there disadvantages/issus which we are aware of ? -Hermanni]
318    
319    
320  4. Xanalogical storage  4. Xanalogical storage
321  ======================  ======================
# Line 321  XXX Line 333  XXX
333  =============  =============
334    
335  Clearly, for block storage to be useful, there has to be a way to  Clearly, for block storage to be useful, there has to be a way to
336  efficiently update documents. We archieve this by a combination of  efficiently update documents/maintain different versions of documents.
337  two mechanisms. Firstly, a *pointer* is an updatable reference to a block;  We achieve this by a combination of two mechanisms. Firstly, a
338    *pointer* is an updatable reference to a block;
339  pointers can be updated by creating a specific kind of Storm block  pointers can be updated by creating a specific kind of Storm block
340  representing an assertion of the form, "pointer ``P`` now points  representing an assertion of the form, "pointer ``P`` now points
341  to block ``B``." Pointers are resolved with the help of a Storm index  to block ``B``." Pointers are resolved with the help of a Storm index
342  mapping pointer identifiers to blocks providing targets for that pointer.  mapping pointer identifiers to blocks providing targets for that pointer.
343  Through this mechanism, we can keep old versions of documents  Through this mechanism, we can keep old versions of documents
344  along with the current versions.  along with the current versions.
345    [Figure ? -Hermanni]
346    
347  Secondly, in the spirit of version control systems like CVS,  Secondly, in the spirit of version control systems like CVS,
348  we do not store each version, but only the differences between versions.  we do not store *each version*, but only the differences between versions.
349  However, we still refer to each full version by the id of a block  However, we still refer to each full version by the id of a block
350  containing that version, even though we do not store this block.  containing that version, even though we do not store this block.
351  When we want to access a particular version, we reconstruct it  When we want to access a particular version, we reconstruct it
352  using the differences, and then check the result using  using the differences, and then check the result using
353  the cryptographic hash in the full version's block id.  the cryptographic hash in the full version's block id.
354    [Figure ? -Hermanni]
355    
356  6.1. Pointers  6.1. Pointers
357  -------------  -------------
# Line 388  item's and peer's keys are each other. T Line 402  item's and peer's keys are each other. T
402  is structured. On the other hand, the overlay connectivity graph of broadcasting  is structured. On the other hand, the overlay connectivity graph of broadcasting
403  approach is formed more or less (depends on implementation) in a random manner.  approach is formed more or less (depends on implementation) in a random manner.
404    
405  When performing queries, in broadcasting approach peer sends a query request to a  When performing queries, in broadcasting approach, peer sends a query request to a
406  subset of its neighbors and these peers to their subsequent neighbors. The  subset of its neighbors and these peers to their subsequent neighbors. The
407  process will continue as long as query's time-to-live (TTL) hasn't been reached.  process will continue as long as query's time-to-live (TTL) value hasn't been reached.
408  In DHT approach, query request is deterministically routed towards the peer  In DHT approach, query request is deterministically routed towards the peer
409  which hosts a specific data item. Routing is based on 'hints' (based on  which hosts a specific data item. Routing is based on 'hints' (based on
410  differences between data item's key and peer's key), which each peer provides  differences between data item's key and peer's key), which each peer provides
# Line 451  into the article as an open research que Line 465  into the article as an open research que
465    
466  Future directions: of course, we shoul implement a prototype  Future directions: of course, we shoul implement a prototype
467    
468  Open issue/Future directions: implement multisource downloading  Open issue/Future directions: implement multisource downloading
469    
470    Future directions: Implement home node model or directory model ?
471    
472  9. Conclusions  9. Conclusions
473  ==============  ==============

Legend:
Removed from v.1.60  
changed lines
  Added in v.1.61

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26