/[gzz]/manuscripts/storm/article.rst
ViewVC logotype

Diff of /manuscripts/storm/article.rst

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.152 by benja, Sat Feb 15 10:52:18 2003 UTC revision 1.153 by benja, Sat Feb 15 11:34:59 2003 UTC
# Line 9  In this paper, we define data mobility a Line 9  In this paper, we define data mobility a
9  movement of documents between computers, different locations  movement of documents between computers, different locations
10  on one computer and movement of content between documents.  on one computer and movement of content between documents.
11  We identify dangling links and alternative versions as major  We identify dangling links and alternative versions as major
12  obstacles for the free movement of data. This paper presents a Storm  obstacles for the free movement of data. This paper presents the Storm
13  (STORage Module) design as one possible solution to the problems in  (STORage Module) design as one possible solution to the problems in
14  the above scenarios. Storm uses location-independent globally unique  the above scenarios. Storm uses location-independent globally unique
15  identifiers, immutable block storage and peer-to-peer networking to  identifiers, append-and-delete-only storage and peer-to-peer networking to
16  resolve problems raised by data mobility. Moreover, we discuss some  resolve problems raised by data mobility. Moreover, we discuss some
17  specific use scenarios related to ad hoc networks, unreliable network  specific use scenarios related to ad hoc networks, unreliable network
18  connections and mobile computing, in which the need for data mobility  connections and mobile computing, in which the need for data mobility
19  is obvious.  is obvious. Our current prototype implementation works on a single system;
20    peer-to-peer networking is in an early prototype stage.
21    
22    
23  Keywords: hypermedia, versioned hypermedia, dangling links, xanalogical storage, P2P,  Keywords: hypermedia, versioned hypermedia, dangling links, xanalogical storage, P2P,
# Line 45  This, we believe, may be the most import Line 46  This, we believe, may be the most import
46  research with regard to hypermedia.  research with regard to hypermedia.
47    
48  In this paper, we examine how location-independent identifiers can  In this paper, we examine how location-independent identifiers can
49  support *data mobility*. Our motivation has been inspired by today's computing  support *data mobility*. Documents often move quite freely
50  world, in which documents move quite freely between computers: they are sent as  between computers: they are sent as
51  e-mail attachments, carried around on disks, published on the web, moved  e-mail attachments, carried around on disks, published on the web, moved
52  between desktop and laptop systems, downloaded for off-line reading or  between desktop and laptop systems, downloaded for off-line reading or
53  copied between computers in a LAN. We use 'data mobility' as a collective  copied between computers in a LAN. We use 'data mobility' as a collective
# Line 62  We address two issues raised by data mob Line 63  We address two issues raised by data mob
63  Dangling links and keeping track of alternative versions.  Dangling links and keeping track of alternative versions.
64  Resolvable location-independent identifiers  Resolvable location-independent identifiers
65  make these issues much easier to deal with, since data  make these issues much easier to deal with, since data
66  can be identified wherever it is moved [#]_.  can be identified wherever it is moved [#]_.
67    Current systems dealing with these issues
68    often do not deal well with many data mobility.
69    
70  .. [#] It might be more appropriate to speak about *resources*  .. [#] It might be more appropriate to speak about *resources*
71     and *references* instead of *documents* and *links*, but     and *references* instead of *documents* and *links*, but
# Line 111  the modified version (e.g., a manual lic Line 114  the modified version (e.g., a manual lic
114  or when a group of people collaborate on a set of documents,  or when a group of people collaborate on a set of documents,
115  synchronizing irregularly with a central server (as in CVS [cvs]_),  synchronizing irregularly with a central server (as in CVS [cvs]_),
116  a network of servers (as in Lotus Notes) or directly with each other  a network of servers (as in Lotus Notes) or directly with each other
117  (as in Groove[?] [ref]). In each of these cases, a user should be able  (as in Groove [groovesurl]_). In each of these cases, a user should be able
118  to work on the version at hand and then either merge it with others  to work on the version at hand and then either merge it with others
119  or fork to a different branch.  or fork to a different branch.
120    
121  In this paper, we present Storm (for *STOrage Module*), a design  In this paper, we present Storm (for *STORage Module*), a design
122  dealing with versioning and dangling links. Storm is a library  dealing with versioning and dangling links. Storm is a library
123  for storing and retrieving data as *blocks*, immutable  for storing and retrieving data as *blocks*, immutable
124  byte sequences identified by cryptographic content hashes  byte sequences identified by cryptographic content hashes
# Line 123  byte sequences identified by cryptograph Line 126  byte sequences identified by cryptograph
126  for versioned data and Xanalogical storage [ted-xanalogical-structure-needed]_.  for versioned data and Xanalogical storage [ted-xanalogical-structure-needed]_.
127  We address the mobility of documents by block storage  We address the mobility of documents by block storage
128  and versioning, while we use Xanalogical storage  and versioning, while we use Xanalogical storage
129  to address the movement of content between documents (copy&paste).  to address the movement of content between documents (copy&paste);
130    see Fig. [ref-storm_layers]_.
 The main contribution of this paper is the Storm design,  
 a hypermedia system built to use the emerging  
 peer-to-peer data lookup technologies to enhance data mobility.  
 Additionally, we hope to  
 provide an input to the ongoing discussion about peer-to-peer  
 hypermedia systems  
 [thompson01coincidence-andalso-bouvin02open-andalso-p2p-hypertext-panel-andalso-lukka02guids]_.  
 Currently, Storm is partially implemented as a part of the Gzz  
 project [gzz]_, which uses Storm exclusively for all disk storage.  
 Storm's peer-to-peer functionality is in a very early stage and not  
 usable yet.  
   
 This paper is structured as follows. In the next section, we describe  
 related work. In section 3, we give an overview of xanalogical model.  
 In section 4, we introduce the basic storage unit of our  
 system, i.e. file-like blocks identified by cryptographic hashes. In section 5,  
 we discuss application-specific reverse indexing of blocks by their  
 content, essential for many applications. In section 6, we present  
 techiques for efficient versioned storage of mutable data on top of blocks.  
 In section 7, we report on implementation experience and future directions.  
 Section 8 concludes the paper.  
131    
132  .. uml:: storm_layers  .. uml:: storm_layers
133      :caption: The Storm model      :caption: The Storm model
# Line 177  Section 8 concludes the paper. Line 159  Section 8 concludes the paper.
159      Pointers.c = z1 + (15, 25);      Pointers.c = z1 + (15, 25);
160      Diffs.c = z1 - (15, 15);      Diffs.c = z1 - (15, 15);
161    
162  .. [Use cases in intro, discussion on how Storm applies to them  The main contribution of this paper is the Storm design,
163     to the end/in Conclusions.]  a hypermedia system built to use the emerging
164    peer-to-peer data lookup technologies to enhance data mobility.
165    Additionally, we hope to
166    provide an input to the ongoing discussion about peer-to-peer
167    hypermedia systems
168    [thompson01coincidence-andalso-bouvin02open-andalso-p2p-hypertext-panel-andalso-lukka02guids]_.
169    Currently, Storm is partially implemented as a part of the Gzz
170    project [gzz]_, which uses Storm exclusively for all disk storage.
171    Storm's peer-to-peer functionality is in a very early stage and not
172    usable yet.
173    
174    This paper is structured as follows. In the next section, we describe
175    related work. In section 3, we give an overview of xanalogical model.
176    In section 4, we introduce the basic storage unit of our
177    system, i.e. file-like blocks identified by cryptographic hashes. In section 5,
178    we discuss application-specific reverse indexing of blocks by their
179    content, essential for many applications. In section 6, we present
180    techiques for efficient versioned storage of mutable data on top of blocks.
181    In section 7, we report on implementation experience and future directions.
182    Section 8 concludes the paper.
183    
184    
185  2. Related Work  2. Related Work
# Line 246  no matter which peer in the network they Line 247  no matter which peer in the network they
247  Version control systems like CVS or RCS [tichy85rcs]_ usually assume  Version control systems like CVS or RCS [tichy85rcs]_ usually assume
248  a central server hosting a repository. The WebDAV/DeltaV protocols,  a central server hosting a repository. The WebDAV/DeltaV protocols,
249  designed for interoperability between version control systems, inherit  designed for interoperability between version control systems, inherit
250  this assumption [rfc2518-andalso-rfc3253]_. On the other hand, Arch [arch]_ places all repositories  this assumption [rfc2518-andalso-rfc3253]_.
251    On the other hand, Arch [arch]_ places all repositories
252  into a global namespace and allows independent developers  into a global namespace and allows independent developers
253  to branch and merge overlapping repositories without any central control  to branch and merge overlapping repositories without any central control
254  [is there a specific ref for this?].  [is there a specific ref for this?].
# Line 399  having to set up a webspace account befo Line 401  having to set up a webspace account befo
401    
402  In the end, some peers will necessarily be more equal than others:  In the end, some peers will necessarily be more equal than others:
403  Published data will be hosted on servers  Published data will be hosted on servers
404  which are permanently on-line, but act as ordinary peers  which are permanently on-line, but are otherwise ordinary peers
405  in the indexing overlay network.  in the indexing overlay network.
406        
407        
# Line 407  in the indexing overlay network. Line 409  in the indexing overlay network.
409  ==================================  ==================================
410    
411  In the xanalogical storage model [ted-xu-tech]_,  In the xanalogical storage model [ted-xu-tech]_,
412  pioneered by the unfinished Project Xanadu [ted-xu-tech]_,  pioneered by the unfinished Project Xanadu,
413  links are not between documents, but individual characters.  links are not between documents, but individual characters.
414  When a character is first typed in, it acquires a permanent id  When a character is first typed in, it acquires a permanent id
415  ("the character 'D' typed by Janne Kujala on 10/8/97 8:37:18"),  ("the character 'D' typed by Janne Kujala on 10/8/97 8:37:18"),
# Line 497  applicable as well: e.g. when working on Line 499  applicable as well: e.g. when working on
499  may be beneficial for the members of the group to see other members'  may be beneficial for the members of the group to see other members'
500  comments of articles etc.  comments of articles etc.
501    
502  Figure XYZ illustrates how xanalogical storage addresses the issue of  Figure [ref-figdocmovement]_ illustrates how xanalogical storage addresses the issue of
503  movement of data between documents. Initially, there are documents D1 and  movement of data between documents. Initially, there are documents D1 and
504  D2, with two links (directed arrows in the figure) from D1 to two different  D2, with two links (directed arrows in the figure) from D1 to two different
505  elements in D2, A and B. The links actually are to the /spans/ A and B that  elements in D2, A and B. The links actually are to the /spans/ A and B that
# Line 1036  between version ``C`` and ``D``, it does Line 1038  between version ``C`` and ``D``, it does
1038  to be altered, because it refers to *version* ``C`` and not  to be altered, because it refers to *version* ``C`` and not
1039  the difference to ``C`` from ``B`` (as in the simplistic scheme).  the difference to ``C`` from ``B`` (as in the simplistic scheme).
1040    
 .. [XXX Figure: diffs from a->b, b->c, c->d; we replace  
    the diffs a->b and b->c by a single diff a->c.]  
   
1041  We can also store the block containing version ``D``  We can also store the block containing version ``D``
1042  in addition to storing the versions above. Then, we can reconstruct  in addition to storing the versions above. Then, we can reconstruct
1043  version ``C`` in two ways: By using the diffs from ``A`` to ``B``  version ``C`` in two ways: By using the diffs from ``A`` to ``B``
# Line 1049  of the diff from ``C`` to ``D`` to versi Line 1048  of the diff from ``C`` to ``D`` to versi
1048     that can be 'skipped' will have to be much higher     that can be 'skipped' will have to be much higher
1049     for this mechanism to be useful.     for this mechanism to be useful.
1050    
 .. [XXX fig?]  
   
1051  Our current implementation is a layer above Storm block storage  Our current implementation is a layer above Storm block storage
1052  and indexing. This layer implements a ``load(version-id) -> version``  and indexing. This layer implements a ``load(version-id) -> version``
1053  interface through the following simplified algorithm:  interface through the following simplified algorithm:
# Line 1123  would have to be sent through the networ Line 1120  would have to be sent through the networ
1120    
1121    
1122  7. Discussion  7. Discussion
1123  ===================================  =============
1124    
1125  In this section we evaluate Storm's design with regard to previously presented  In this section we evaluate Storm's design with regard to previously presented
1126  use cases, summarize Storm's implementation status and outline open issues  use cases, summarize Storm's implementation status and outline open issues
1127  and future work.  and future work.
1128    
1129    
1130  7.1 Evaluation  7.1 Evaluation
1131  --------------  --------------
1132    
# Line 1142  location-independent. In a peer-to-peer Line 1141  location-independent. In a peer-to-peer
1141  id returns a location where the data is currently available. If the  id returns a location where the data is currently available. If the
1142  publisher removes the document permanently, but it is archived elsewhere,  publisher removes the document permanently, but it is archived elsewhere,
1143  the archives act as peers similarly. When there is no network connection  the archives act as peers similarly. When there is no network connection
1144  available, but a local copy instead, the lookup points to that. If a  available, but a local copy instead, Storm can find it. If a
1145  document and a link to it are received independently, e.g. as attachments in  document and a link to it are received independently, e.g. as attachments in
1146  separate e-mails, or a link to a document in the local intranet is e-mailed,  separate e-mails, or a link to a document in the local intranet is e-mailed,
 (all those links work :o .. this was also discussed in intro already (index))  
1147  When people meet live, e.g. on a train, and form an ad-hoc network, they are  When people meet live, e.g. on a train, and form an ad-hoc network, they are
1148  able to see each other's documents and follow links to them if a  able to see each other's public documents and follow links to them if a
1149  peer-to-peer implementation of Storm is used (and pools published  peer-to-peer implementation of Storm is used.
 accordingly..? what should be discussed here?).  
1150  Finally, if a document is split to parts (or content from one copy-pasted to  Finally, if a document is split to parts (or content from one copy-pasted to
1151  another), links to the elements that are then in the new documents do not  another), links to the elements that are then in the new documents do not
1152  break, bacause with xanalogical storage the links are actually to spans that  break, bacause with xanalogical storage the links are actually to spans that
1153  are transcluded in all the documents that show them (as illustrated in  are transcluded in all the documents that show them (as illustrated in
1154  figure XYZ in section 3, xanalogical storage).  figure [ref-figdocmovement]_ in section 3).
1155    
1156    .. (all those links work :o .. this was also discussed in intro already (index))
1157    
1158  *Tracking alternative versions*. Because Storm utilises immutable blocks,  *Tracking alternative versions*. Because Storm utilises immutable blocks,
1159  each modification to a document creates a new block. When a document is  each modification to a document creates a new block. When a document is
1160  modified on several independent, unconnected systems, if there are  modified on several independent, unconnected systems, if there are
# Line 1187  trivial? Line 1186  trivial?
1186  7.2 Status, open issues and future work  7.2 Status, open issues and future work
1187  ---------------------------------------  ---------------------------------------
1188    
   
1189  [worth to mention, security in general ? -Hermanni]  [worth to mention, security in general ? -Hermanni]
1190  1) Open issue: We want to be able  1) Open issue: We want to be able
1191  to copy the blocks from the NYT publication pool to our local computer  to copy the blocks from the NYT publication pool to our local computer

Legend:
Removed from v.1.152  
changed lines
  Added in v.1.153

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26