/[gzz]/manuscripts/storm/article.rst
ViewVC logotype

Diff of /manuscripts/storm/article.rst

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.84 by antont, Tue Feb 4 12:32:19 2003 UTC revision 1.85 by hemppah, Tue Feb 4 14:08:42 2003 UTC
# Line 13  in light of use cases from a realistic(? Line 13  in light of use cases from a realistic(?
13  future work in areas such as interoperability, scalability and security is  future work in areas such as interoperability, scalability and security is
14  identified.  identified.
15    
16  Keywords: data mobility, global unique identifiers, hypermedia management  Keywords: data mobility, ad-hoc data mobility, mobile hypermedia,
17  system, hyperbase, open hypermedia, structural computing, xanalogical media,  global unique identifiers, hypermedia management system, hyperbase,
18  permanent storage systems, peer-to-peer networking, distributed hash table,  open hypermedia, structural computing, xanalogical media, permanent
19  ad-hoc data mobility, mobile hypermedia  storage systems, peer-to-peer networking, structured overlay networks,
20    location inpenpendent routing, distributed hash table
21    
22    
23    
24  1. Introduction  1. Introduction
# Line 288  as *blocks*, byte sequences identified b Line 290  as *blocks*, byte sequences identified b
290  as regular files, but they are immutable, since any change to the  as regular files, but they are immutable, since any change to the
291  byte sequence would change the hash (and thus create a different block).  byte sequence would change the hash (and thus create a different block).
292  Mutable data structures are built on top of the immutable blocks  Mutable data structures are built on top of the immutable blocks
293  (see Section 6).  (see Section 6).
294    
295  Immutable blocks has several benefits over existing data storing  Immutable blocks has several benefits over existing data storing
296  techiques:  techiques:
# Line 299  stored in different blocks. Line 301  stored in different blocks.
301  [Previous sentence doesn't parse to me (what level ?) :( -Hermanni]  [Previous sentence doesn't parse to me (what level ?) :( -Hermanni]
302  To replicate all data from computer A  To replicate all data from computer A
303  on computer B, it suffices to copy all blocks from A to B that B  on computer B, it suffices to copy all blocks from A to B that B
304  does not already store. On the other hand, several popular database management  does not already store. On the other hand for instance, several popular
305  systems (e.g. Lotus Notes [ref]) have complex replication schemes, which may  database management systems (e.g. Lotus Notes [ref]) have complex
306  led awkward replication conflicts.  replication schemes, which may led awkward replication conflicts,
307    because of they lack the immutable properties of data.
308  [Or does this belong to diff section ? -Hermanni]  [Or does this belong to diff section ? -Hermanni]
309    
310  Storm blocks are MIME messages [ref MIME], i.e., objects with  Storm blocks are MIME messages [ref MIME], i.e., objects with
# Line 317  the following interface:: Line 320  the following interface::
320      add(block)      add(block)
321      delete(block)      delete(block)
322            
 [analogy to regular Hash Table/DHT ? -Hermanni]  
   
323  Implementations may store blocks in RAM, in individual files,  Implementations may store blocks in RAM, in individual files,
324  in a Zip archive, in a database or through other means.  in a Zip archive, in a database or through other means.
325  We have implemented the first three (using hexadecimal  We have implemented the first three (using hexadecimal
# Line 332  will be lost; the previous version of th Line 333  will be lost; the previous version of th
333  be accessible. This makes Storm well suited as a basis  be accessible. This makes Storm well suited as a basis
334  for implementing experimental projects (such as ours).  for implementing experimental projects (such as ours).
335    
336  When used in a network environment, Storm ids do not provide  [Example of Microsoft Word ? -Hermanni]
337  a hint as to where in the network a specific block can be found.  
338  However, current peer-to-peer systems could be used to  When used in a network environment, Storm IDs do not provide
339  find blocks efficiently in a location independent fashion; for example,  a hint as to where a specific block is stored in the network.
340  Freenet [ref], recent Gnutella-based clients (e.g. Shareaza [ref]),  However, many existing peer-to-peer systems could be used to
341  Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers  find arbitrary blocks in a location independent fashion;
342    for example, Freenet [ref], recent Gnutella-based clients
343    (e.g. Shareaza [ref]), Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers
344  [e.g. ref: magnet uri]. Footnote:However, we have not put a network  [e.g. ref: magnet uri]. Footnote:However, we have not put a network
345  implementation into regular use yet and thus can only describe our  implementation into regular use yet and thus can only describe our
346  design, not report on implementation experience.  design, not report on implementation experience.
# Line 356  from the net, without broken links. Line 359  from the net, without broken links.
359  [Previous sentence doesn't parse to me: more simple :( -Hermanni]  [Previous sentence doesn't parse to me: more simple :( -Hermanni]
360    
361  Given a peer-to-peer distribution mechanism, it would be possible  Given a peer-to-peer distribution mechanism, it would be possible
362  to retrieve blocks from any peer online that has a copy  to retrieve blocks from any participating peer online that has a copy
363  in its cache or permanent storage. This is similar to the Squirrel  in its local cache or permanent storage. This is similar to the Squirrel
364  web cache [ref] [more refs? -Hermanni], but does not require trust  web cache [ref] [more refs? -Hermanni], but does not require trust
365  between the peers, since it is possible to check the blocks' integrity by using  between the peers, since it is possible to check the blocks' integrity by using
366  cryptographic hashes. Since much-requested blocks would be  cryptographic hashes, as used in many peer-to-peer applications
367  cached on many systems, such a network could deal with XXX  (e.g. [ref: ed2k/overnet, shareaza]). Since much-requested blocks would be
368    cached on many systems, a network could deal with hotspots
369  much more easily. On the other hand, there are privacy  much more easily. On the other hand, there are privacy
370  concerns with exposing one's browser cache to the outside world.  concerns with exposing one's browser cache to the outside world.
371    
# Line 417  Of course, doing any expensive operation Line 421  Of course, doing any expensive operation
421  in a document does not scale very well. In practice,  in a document does not scale very well. In practice,
422  characters typed in consecutively are given consecutive ids,  characters typed in consecutively are given consecutive ids,
423  such as ``...:4``, ``...:5``, ``...:6`` and so on, and  such as ``...:4``, ``...:5``, ``...:6`` and so on, and
424  operations are on *spans*, consecutive ranges of characters  operations are on *spans*, i.e. consecutive ranges of characters
425  (``...:4-6``). In Storm, in each editor session we create a  (``...:4-6``) in a document. In Storm, in each editor session we
426  block with all characters entered in this session (the content type  create a block with all characters entered in this session (the content type
427  being ``text/plain``). To designate a span of characters  being ``text/plain``). To designate a span of characters
428  from that session, we use the block's id, the offset of the first  from that session, we use the block's id, the offset of the first
429  character, and the number of characters in the span.  character, and the number of characters in the span.
430  This technique was first introduced in [ref ht02 paper].  This technique was first introduced in [ref ht02 paper].
431    
432  In Xanadu, characters are stored to append-only *scrolls*  In Xanadu, characters are stored to append-only *scrolls*
433  when they are typed [ref]. Because of this, we call the blocks  when they are typed [ref]. Because of this, in Storm, we call the
434  containing the actual characters *scroll blocks*. The documents  blocks containing the actual characters *scroll blocks*. The documents
435  do not actually contain the characters; instead, they are  do not actually contain the characters; instead, they are
436  *virtual files* containing span references as described above.  *virtual files* containing span references as described above.
437  To show a document, the scroll blocks it references are loaded  To show a document, the scroll blocks it references are loaded
# Line 459  that this will not be a major scalabilit Line 463  that this will not be a major scalabilit
463  systems that allow range queries, such as skip graphs [ref]  systems that allow range queries, such as skip graphs [ref]
464  and skipnet [ref], may prove useful.  and skipnet [ref], may prove useful.
465    
466  [This might be relevant:  [This might be relevant also:
467  http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf  http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf
468  -Hermanni]  -Hermanni]
469    
# Line 481  of Storm blocks: finding a block based o Line 485  of Storm blocks: finding a block based o
485  Storm provides a general API for indexing blocks in  Storm provides a general API for indexing blocks in
486  application-specific ways. We have implemented indexing  application-specific ways. We have implemented indexing
487  on a local machine, but the interface is designed so that  on a local machine, but the interface is designed so that
488  implementation on top of a distributed hashtable  implementation on top of networking overlay (e.g. distributed hashtable)
489  will be trivial.  will be trivial.
490    
491  [Benja, this might be useful for defining Storm APIs for DHTs etc:  [Benja, this might be useful for defining Storm APIs for DHTs etc:
# Line 493  mappings into the index. Instead, applic Line 497  mappings into the index. Instead, applic
497  to index blocks provide the following callback  to index blocks provide the following callback
498  to a Storm pool::  to a Storm pool::
499    
500    [What is mapping ? We should explain this :) -Hermanni]
501    
502      getMappings(block) -> set of (key, value) pairs      getMappings(block) -> set of (key, value) pairs
503    
504  This callback processes a block and returns a set of mappings  This callback processes a block and returns a set of mappings
# Line 513  to e.g. remove associated mappings when Line 519  to e.g. remove associated mappings when
519  As an example, the ``getMappings()`` method of our Xanalogical  As an example, the ``getMappings()`` method of our Xanalogical
520  storage implementation will, for a block containing a document,  storage implementation will, for a block containing a document,
521  collect all the spans in a document, and return mappings  collect all the spans in a document, and return mappings
522  from their scroll blocks' ids to the spans and their positions  from their scroll blocks' IDs to the spans and their positions
523  in the document. When we want to find the transclusions of  in the document. When we want to find the transclusions of
524  a span, we ``get()`` the mappings for the id of that span's  a span, we use ``get()`` to get the mappings for the ID of
525  scroll block, and load the document blocks referenced  that span's scroll block, and load the document blocks referenced
526  by the mappings.  by the mappings.
527    
528  In a networked implementation, each peer is responsible  In a networked implementation, each peer is responsible

Legend:
Removed from v.1.84  
changed lines
  Added in v.1.85

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26