/[gzz]/manuscripts/storm/article.rst
ViewVC logotype

Diff of /manuscripts/storm/article.rst

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.133 by antont, Wed Feb 12 12:39:34 2003 UTC revision 1.134 by hemppah, Wed Feb 12 13:24:41 2003 UTC
# Line 148  Storm's peer-to-peer functionality is in Line 148  Storm's peer-to-peer functionality is in
148  usable yet.  usable yet.
149    
150  This paper is structured as follows. In the next section, we describe  This paper is structured as follows. In the next section, we describe
151  related work. In section 3, we introduce the basic storage unit of our  related work. In section 3, we give an overview of xanalogical model.
152  system, file-like blocks identified by cryptographic hashes.  In section 4, we introduce the basic storage unit of our
153  In section 4, we discuss our implementation of Xanalogical storage  system, i.e. file-like blocks identified by cryptographic hashes. In section 5,
154  on top of the block system. In section 5, we discuss application-specific  we discuss application-specific reverse indexing of blocks by their
155  reverse indexing of blocks by their content, essential for many applications.  content, essential for many applications. In section 6, we present
156  In section 6, we present techiques for efficient versioned storage  techiques for efficient versioned storage of mutable data on top of blocks.
157  of mutable data on top of blocks. In section 7,  In section 7, we report on implementation experience and future directions.
 we report on implementation experience and future directions.  
158  Section 8 concludes the paper.  Section 8 concludes the paper.
159    
160  .. uml:: storm_layers  .. uml:: storm_layers
# Line 419  Published data will be hosted on servers Line 418  Published data will be hosted on servers
418  which are permanently on-line, but act as ordinary peers  which are permanently on-line, but act as ordinary peers
419  in the indexing overlay network.  in the indexing overlay network.
420        
421      
422    3. Overview of Xanalogical storage
423    ==================================
424    
425    In the xanalogical storage model [ref],
426    pioneered by the unfinished Project Xanadu [ref],
427    links are not between documents, but individual characters.
428    When a character is first typed in, it acquires a permanent id
429    ("the character 'D' typed by Janne Kujala on 10/8/97 8:37:18"),
430    which it retains when copied to a different document, distinguishing
431    it from all similar characters typed in independently [#]_.
432    A link is shown between any two documents containing the characters
433    that the link connects. Xanalogical links are external and bidirectional.
434    
435    .. [#] Xanalogical storage is not limited to text. We speak about
436       *characters* because it simplifies the explanation; picture's pixels
437       or frames of video could be substituted.
438    
439  3. Block storage  In addition to content links, xanalogical storage keeps an index of
440  ================  transclusions: identical characters copied into different documents.
441    Through this mechanism, the system can show to the user all documents
442    that share text with the current document.
443    
444    To keep track of links and transclusions, the system keeps a global index
445    of documents by the characters they contain, and of links by the characters
446    they refer to. Thus, for each character in the document, the system
447    queries the index for other documents containing this character,
448    and shows them as transclusions. Resolving links is a multi-step process.
449    Each link is modeled as two collections of characters: the two
450    endpoints of the link. To show links to a specific document,
451    the system firstly uses the link index to find links
452    to each character in the document. Secondly, for each link,
453    it looks at the *other* set of characters in the link-- the target
454    of the link, if the original character was the source, and vice versa.
455    Thirdly, it looks for documents containing these target characters.
456    This way, even if both the source and target characters of the link
457    are moved to a different document, the link stays connected to them.
458    
459    Of course, doing any expensive operation for *every* character
460    in a document does not scale very well. In practice,
461    characters typed in consecutively are given consecutive ids,
462    such as ``...:4``, ``...:5``, ``...:6`` and so on, and
463    operations are on *spans*, i.e. consecutive ranges of characters
464    (``...:4-6``) in a document. In Storm, in each editor session we
465    create a block with all characters entered in this session (the content type
466    being ``text/plain``). To designate a span of characters
467    from that session, we use the block's id, the offset of the first
468    character, and the number of characters in the span.
469    This technique was first introduced in [ref ht02 paper].
470    
471    In Xanadu, characters are stored to append-only *scrolls*
472    when they are typed [ref]. Because of this, in Storm, we call the
473    blocks containing the actual characters *scroll blocks*. The documents
474    do not actually contain the characters; instead, they are
475    *virtual files* containing span references as described above.
476    To show a document, the scroll blocks it references are loaded
477    and the characters retrieved from there [#]_.
478    
479    .. [#] It is unclear whether this approach is efficient for text
480       in the Storm framework; in the future, we may try storing
481       the characters in the documents themselves, along with their
482       permanent identifiers. For images or video, on the other hand,
483       it is clearly beneficial if content appearing in different
484       documents-- or different versions of a document-- is only
485       stored once, in a block only referred to wherever
486       the data is transcluded.
487    
488    
489    
490    4. Storm block storage
491    ======================
492    
493  In Storm, all data is stored  In Storm, all data is stored
494  as *blocks*, byte sequences identified by a SHA-1  as *blocks*, byte sequences identified by a SHA-1
# Line 571  for the experimental Gzz system, a platf Line 638  for the experimental Gzz system, a platf
638  to overcome the limitations of traditional file-based applications.  to overcome the limitations of traditional file-based applications.
639    
640    
641  3.1. Implementation  4.1. Implementation
642  -------------------  -------------------
643    
644  Storm blocks are MIME messages [ref MIME], i.e., objects with  Storm blocks are MIME messages [ref MIME], i.e., objects with
# Line 641  version of a document whose identifier i Line 708  version of a document whose identifier i
708  the software (mutable documents are described in section 6.1).  the software (mutable documents are described in section 6.1).
709    
710    
 4. Xanalogical storage  
 ======================  
   
 In the xanalogical storage model [ref],  
 pioneered by the unfinished Project Xanadu [ref],  
 links are not between documents, but individual characters.  
 When a character is first typed in, it acquires a permanent id  
 ("the character 'D' typed by Janne Kujala on 10/8/97 8:37:18"),  
 which it retains when copied to a different document, distinguishing  
 it from all similar characters typed in independently [#]_.  
 A link is shown between any two documents containing the characters  
 that the link connects. Xanalogical links are external and bidirectional.  
   
 .. [#] Xanalogical storage is not limited to text. We speak about  
    *characters* because it simplifies the explanation; picture's pixels  
    or frames of video could be substituted.  
   
 In addition to content links, xanalogical storage keeps an index of  
 transclusions: identical characters copied into different documents.  
 Through this mechanism, the system can show to the user all documents  
 that share text with the current document.  
   
 To keep track of links and transclusions, the system keeps a global index  
 of documents by the characters they contain, and of links by the characters  
 they refer to. Thus, for each character in the document, the system  
 queries the index for other documents containing this character,  
 and shows them as transclusions. Resolving links is a multi-step process.  
 Each link is modeled as two collections of characters: the two  
 endpoints of the link. To show links to a specific document,  
 the system firstly uses the link index to find links  
 to each character in the document. Secondly, for each link,  
 it looks at the *other* set of characters in the link-- the target  
 of the link, if the original character was the source, and vice versa.  
 Thirdly, it looks for documents containing these target characters.  
 This way, even if both the source and target characters of the link  
 are moved to a different document, the link stays connected to them.  
   
 Of course, doing any expensive operation for *every* character  
 in a document does not scale very well. In practice,  
 characters typed in consecutively are given consecutive ids,  
 such as ``...:4``, ``...:5``, ``...:6`` and so on, and  
 operations are on *spans*, i.e. consecutive ranges of characters  
 (``...:4-6``) in a document. In Storm, in each editor session we  
 create a block with all characters entered in this session (the content type  
 being ``text/plain``). To designate a span of characters  
 from that session, we use the block's id, the offset of the first  
 character, and the number of characters in the span.  
 This technique was first introduced in [ref ht02 paper].  
   
 In Xanadu, characters are stored to append-only *scrolls*  
 when they are typed [ref]. Because of this, in Storm, we call the  
 blocks containing the actual characters *scroll blocks*. The documents  
 do not actually contain the characters; instead, they are  
 *virtual files* containing span references as described above.  
 To show a document, the scroll blocks it references are loaded  
 and the characters retrieved from there [#]_.  
   
 .. [#] It is unclear whether this approach is efficient for text  
    in the Storm framework; in the future, we may try storing  
    the characters in the documents themselves, along with their  
    permanent identifiers. For images or video, on the other hand,  
    it is clearly beneficial if content appearing in different  
    documents-- or different versions of a document-- is only  
    stored once, in a block only referred to wherever  
    the data is transcluded.  
   
 Our current implementation shows only links between documents  
 that are in memory at the same time [screenshot of xupdf, perhaps ref too  
 (submitted) antont: was thinking the same, it would illustrate this well].  
 In the future, we will implement a global distributed index at top of  
 a distributed hashtable, with the scroll blocks' ids as the keys.  
 To find the transclusions of a span, the system will retrieve  
 all transclusions of any span in the scroll block, then  
 sort out those that do not overlap the span in question.  
   
 Since the problem is to search for overlapping ranges,  
 the spans cannot be used as hashtable keys. However, as the blocks  
 will be relatively small (limited by the amount of text  
 the user enters between two saves of a document), we hope  
 that this will not be a major scalability problem. Otherwise,  
 systems that allow range queries, such as skip graphs [ref]  
 and skipnet [ref], may prove useful.  
   
 .. [how does video work here, i.e. are they huge blocks or collections of many  
    small, frames? or a sequence between two keyframes?]  
   
    Benja says: Probably large, but the number of links to them  
    still won't be that big, I guess. Not sure how to discuss this  
    in the text; feel free to propose something :-)  
   
    Toni: will try.. wondering also about who to consult :o  
   
 .. [This might be relevant also:  
    http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf  
    -Hermanni]  
   
 One question raised by xanalogical storage is which links to show  
 for a popular document that has been linked to by many users.  
 We hope to address this problem by collaborative filtering  
 of links [explain, ref (grouplens.org pubs?)]. There has been research on  
 collaborative filtering in peer-to-peer systems  
 without compromising participants' privacy [ref John Canny].  
 For some purposes simple rules based on e.g. belonging to a group may be  
 applicable as well: e.g. when working on a project with a project group, it  
 may be beneficial for the members of the group to see other members'  
 comments of articles etc.  
   
   
711  5. Application-specific reverse indexing  5. Application-specific reverse indexing
712  ========================================  ========================================
713    
# Line 1166  names, which is again outside the scope Line 1125  names, which is again outside the scope
1125     Comments may be new entities(?) linking to it       Comments may be new entities(?) linking to it  
1126    
1127    
1128    
1129    Our current implementation shows only links between documents
1130    that are in memory at the same time [screenshot of xupdf, perhaps ref too
1131    (submitted) antont: was thinking the same, it would illustrate this well].
1132    In the future, we will implement a global distributed index at top of
1133    a distributed hashtable, with the scroll blocks' ids as the keys.
1134    To find the transclusions of a span, the system will retrieve
1135    all transclusions of any span in the scroll block, then
1136    sort out those that do not overlap the span in question.
1137    
1138    Since the problem is to search for overlapping ranges,
1139    the spans cannot be used as hashtable keys. However, as the blocks
1140    will be relatively small (limited by the amount of text
1141    the user enters between two saves of a document), we hope
1142    that this will not be a major scalability problem. Otherwise,
1143    systems that allow range queries, such as skip graphs [ref]
1144    and skipnet [ref], may prove useful.
1145    
1146    .. [how does video work here, i.e. are they huge blocks or collections of many
1147       small, frames? or a sequence between two keyframes?]
1148    
1149       Benja says: Probably large, but the number of links to them
1150       still won't be that big, I guess. Not sure how to discuss this
1151       in the text; feel free to propose something :-)
1152    
1153       Toni: will try.. wondering also about who to consult :o
1154    
1155    .. [This might be relevant also:
1156       http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf
1157       -Hermanni]
1158    
1159    One question raised by xanalogical storage is which links to show
1160    for a popular document that has been linked to by many users.
1161    We hope to address this problem by collaborative filtering
1162    of links [explain, ref (grouplens.org pubs?)]. There has been research on
1163    collaborative filtering in peer-to-peer systems
1164    without compromising participants' privacy [ref John Canny].
1165    For some purposes simple rules based on e.g. belonging to a group may be
1166    applicable as well: e.g. when working on a project with a project group, it
1167    may be beneficial for the members of the group to see other members'
1168    comments of articles etc.
1169      
1170      
1171  Open issue: Tree hashing, also possibly needed for (OceanStore-like?)  Open issue: Tree hashing, also possibly needed for (OceanStore-like?)
1172  distribution of shares  distribution of shares
1173    

Legend:
Removed from v.1.133  
changed lines
  Added in v.1.134

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26