9 |
movement of documents between computers, different locations |
movement of documents between computers, different locations |
10 |
on one computer and movement of content between documents. |
on one computer and movement of content between documents. |
11 |
We identify dangling links and alternative versions as major |
We identify dangling links and alternative versions as major |
12 |
obstacles for the free movement of data. This paper presents a Storm |
obstacles for the free movement of data. This paper presents the Storm |
13 |
(STORage Module) design as one possible solution to the problems in |
(STORage Module) design as one possible solution to the problems in |
14 |
the above scenarios. Storm uses location-independent globally unique |
the above scenarios. Storm uses location-independent globally unique |
15 |
identifiers, immutable block storage and peer-to-peer networking to |
identifiers, append-and-delete-only storage and peer-to-peer networking to |
16 |
resolve problems raised by data mobility. Moreover, we discuss some |
resolve problems raised by data mobility. Moreover, we discuss some |
17 |
specific use scenarios related to ad hoc networks, unreliable network |
specific use scenarios related to ad hoc networks, unreliable network |
18 |
connections and mobile computing, in which the need for data mobility |
connections and mobile computing, in which the need for data mobility |
19 |
is obvious. |
is obvious. Our current prototype implementation works on a single system; |
20 |
|
peer-to-peer networking is in an early prototype stage. |
21 |
|
|
22 |
|
|
23 |
Keywords: hypermedia, versioned hypermedia, dangling links, xanalogical storage, P2P, |
Keywords: hypermedia, versioned hypermedia, dangling links, xanalogical storage, P2P, |
46 |
research with regard to hypermedia. |
research with regard to hypermedia. |
47 |
|
|
48 |
In this paper, we examine how location-independent identifiers can |
In this paper, we examine how location-independent identifiers can |
49 |
support *data mobility*. Our motivation has been inspired by today's computing |
support *data mobility*. Documents often move quite freely |
50 |
world, in which documents move quite freely between computers: they are sent as |
between computers: they are sent as |
51 |
e-mail attachments, carried around on disks, published on the web, moved |
e-mail attachments, carried around on disks, published on the web, moved |
52 |
between desktop and laptop systems, downloaded for off-line reading or |
between desktop and laptop systems, downloaded for off-line reading or |
53 |
copied between computers in a LAN. We use 'data mobility' as a collective |
copied between computers in a LAN. We use 'data mobility' as a collective |
63 |
Dangling links and keeping track of alternative versions. |
Dangling links and keeping track of alternative versions. |
64 |
Resolvable location-independent identifiers |
Resolvable location-independent identifiers |
65 |
make these issues much easier to deal with, since data |
make these issues much easier to deal with, since data |
66 |
can be identified wherever it is moved [#]_. |
can be identified wherever it is moved [#]_. |
67 |
|
Current systems dealing with these issues |
68 |
|
often do not deal well with many data mobility. |
69 |
|
|
70 |
.. [#] It might be more appropriate to speak about *resources* |
.. [#] It might be more appropriate to speak about *resources* |
71 |
and *references* instead of *documents* and *links*, but |
and *references* instead of *documents* and *links*, but |
114 |
or when a group of people collaborate on a set of documents, |
or when a group of people collaborate on a set of documents, |
115 |
synchronizing irregularly with a central server (as in CVS [cvs]_), |
synchronizing irregularly with a central server (as in CVS [cvs]_), |
116 |
a network of servers (as in Lotus Notes) or directly with each other |
a network of servers (as in Lotus Notes) or directly with each other |
117 |
(as in Groove[?] [ref]). In each of these cases, a user should be able |
(as in Groove [groovesurl]_). In each of these cases, a user should be able |
118 |
to work on the version at hand and then either merge it with others |
to work on the version at hand and then either merge it with others |
119 |
or fork to a different branch. |
or fork to a different branch. |
120 |
|
|
121 |
In this paper, we present Storm (for *STOrage Module*), a design |
In this paper, we present Storm (for *STORage Module*), a design |
122 |
dealing with versioning and dangling links. Storm is a library |
dealing with versioning and dangling links. Storm is a library |
123 |
for storing and retrieving data as *blocks*, immutable |
for storing and retrieving data as *blocks*, immutable |
124 |
byte sequences identified by cryptographic content hashes |
byte sequences identified by cryptographic content hashes |
126 |
for versioned data and Xanalogical storage [ted-xanalogical-structure-needed]_. |
for versioned data and Xanalogical storage [ted-xanalogical-structure-needed]_. |
127 |
We address the mobility of documents by block storage |
We address the mobility of documents by block storage |
128 |
and versioning, while we use Xanalogical storage |
and versioning, while we use Xanalogical storage |
129 |
to address the movement of content between documents (copy&paste). |
to address the movement of content between documents (copy&paste); |
130 |
|
see Fig. [ref-storm_layers]_. |
|
The main contribution of this paper is the Storm design, |
|
|
a hypermedia system built to use the emerging |
|
|
peer-to-peer data lookup technologies to enhance data mobility. |
|
|
Additionally, we hope to |
|
|
provide an input to the ongoing discussion about peer-to-peer |
|
|
hypermedia systems |
|
|
[thompson01coincidence-andalso-bouvin02open-andalso-p2p-hypertext-panel-andalso-lukka02guids]_. |
|
|
Currently, Storm is partially implemented as a part of the Gzz |
|
|
project [gzz]_, which uses Storm exclusively for all disk storage. |
|
|
Storm's peer-to-peer functionality is in a very early stage and not |
|
|
usable yet. |
|
|
|
|
|
This paper is structured as follows. In the next section, we describe |
|
|
related work. In section 3, we give an overview of xanalogical model. |
|
|
In section 4, we introduce the basic storage unit of our |
|
|
system, i.e. file-like blocks identified by cryptographic hashes. In section 5, |
|
|
we discuss application-specific reverse indexing of blocks by their |
|
|
content, essential for many applications. In section 6, we present |
|
|
techiques for efficient versioned storage of mutable data on top of blocks. |
|
|
In section 7, we report on implementation experience and future directions. |
|
|
Section 8 concludes the paper. |
|
131 |
|
|
132 |
.. uml:: storm_layers |
.. uml:: storm_layers |
133 |
:caption: The Storm model |
:caption: The Storm model |
159 |
Pointers.c = z1 + (15, 25); |
Pointers.c = z1 + (15, 25); |
160 |
Diffs.c = z1 - (15, 15); |
Diffs.c = z1 - (15, 15); |
161 |
|
|
162 |
.. [Use cases in intro, discussion on how Storm applies to them |
The main contribution of this paper is the Storm design, |
163 |
to the end/in Conclusions.] |
a hypermedia system built to use the emerging |
164 |
|
peer-to-peer data lookup technologies to enhance data mobility. |
165 |
|
Additionally, we hope to |
166 |
|
provide an input to the ongoing discussion about peer-to-peer |
167 |
|
hypermedia systems |
168 |
|
[thompson01coincidence-andalso-bouvin02open-andalso-p2p-hypertext-panel-andalso-lukka02guids]_. |
169 |
|
Currently, Storm is partially implemented as a part of the Gzz |
170 |
|
project [gzz]_, which uses Storm exclusively for all disk storage. |
171 |
|
Storm's peer-to-peer functionality is in a very early stage and not |
172 |
|
usable yet. |
173 |
|
|
174 |
|
This paper is structured as follows. In the next section, we describe |
175 |
|
related work. In section 3, we give an overview of xanalogical model. |
176 |
|
In section 4, we introduce the basic storage unit of our |
177 |
|
system, i.e. file-like blocks identified by cryptographic hashes. In section 5, |
178 |
|
we discuss application-specific reverse indexing of blocks by their |
179 |
|
content, essential for many applications. In section 6, we present |
180 |
|
techiques for efficient versioned storage of mutable data on top of blocks. |
181 |
|
In section 7, we report on implementation experience and future directions. |
182 |
|
Section 8 concludes the paper. |
183 |
|
|
184 |
|
|
185 |
2. Related Work |
2. Related Work |
247 |
Version control systems like CVS or RCS [tichy85rcs]_ usually assume |
Version control systems like CVS or RCS [tichy85rcs]_ usually assume |
248 |
a central server hosting a repository. The WebDAV/DeltaV protocols, |
a central server hosting a repository. The WebDAV/DeltaV protocols, |
249 |
designed for interoperability between version control systems, inherit |
designed for interoperability between version control systems, inherit |
250 |
this assumption [rfc2518-andalso-rfc3253]_. On the other hand, Arch [arch]_ places all repositories |
this assumption [rfc2518-andalso-rfc3253]_. |
251 |
|
On the other hand, Arch [arch]_ places all repositories |
252 |
into a global namespace and allows independent developers |
into a global namespace and allows independent developers |
253 |
to branch and merge overlapping repositories without any central control |
to branch and merge overlapping repositories without any central control |
254 |
[is there a specific ref for this?]. |
[is there a specific ref for this?]. |
401 |
|
|
402 |
In the end, some peers will necessarily be more equal than others: |
In the end, some peers will necessarily be more equal than others: |
403 |
Published data will be hosted on servers |
Published data will be hosted on servers |
404 |
which are permanently on-line, but act as ordinary peers |
which are permanently on-line, but are otherwise ordinary peers |
405 |
in the indexing overlay network. |
in the indexing overlay network. |
406 |
|
|
407 |
|
|
409 |
================================== |
================================== |
410 |
|
|
411 |
In the xanalogical storage model [ted-xu-tech]_, |
In the xanalogical storage model [ted-xu-tech]_, |
412 |
pioneered by the unfinished Project Xanadu [ted-xu-tech]_, |
pioneered by the unfinished Project Xanadu, |
413 |
links are not between documents, but individual characters. |
links are not between documents, but individual characters. |
414 |
When a character is first typed in, it acquires a permanent id |
When a character is first typed in, it acquires a permanent id |
415 |
("the character 'D' typed by Janne Kujala on 10/8/97 8:37:18"), |
("the character 'D' typed by Janne Kujala on 10/8/97 8:37:18"), |
499 |
may be beneficial for the members of the group to see other members' |
may be beneficial for the members of the group to see other members' |
500 |
comments of articles etc. |
comments of articles etc. |
501 |
|
|
502 |
Figure XYZ illustrates how xanalogical storage addresses the issue of |
Figure [ref-figdocmovement]_ illustrates how xanalogical storage addresses the issue of |
503 |
movement of data between documents. Initially, there are documents D1 and |
movement of data between documents. Initially, there are documents D1 and |
504 |
D2, with two links (directed arrows in the figure) from D1 to two different |
D2, with two links (directed arrows in the figure) from D1 to two different |
505 |
elements in D2, A and B. The links actually are to the /spans/ A and B that |
elements in D2, A and B. The links actually are to the /spans/ A and B that |
1038 |
to be altered, because it refers to *version* ``C`` and not |
to be altered, because it refers to *version* ``C`` and not |
1039 |
the difference to ``C`` from ``B`` (as in the simplistic scheme). |
the difference to ``C`` from ``B`` (as in the simplistic scheme). |
1040 |
|
|
|
.. [XXX Figure: diffs from a->b, b->c, c->d; we replace |
|
|
the diffs a->b and b->c by a single diff a->c.] |
|
|
|
|
1041 |
We can also store the block containing version ``D`` |
We can also store the block containing version ``D`` |
1042 |
in addition to storing the versions above. Then, we can reconstruct |
in addition to storing the versions above. Then, we can reconstruct |
1043 |
version ``C`` in two ways: By using the diffs from ``A`` to ``B`` |
version ``C`` in two ways: By using the diffs from ``A`` to ``B`` |
1048 |
that can be 'skipped' will have to be much higher |
that can be 'skipped' will have to be much higher |
1049 |
for this mechanism to be useful. |
for this mechanism to be useful. |
1050 |
|
|
|
.. [XXX fig?] |
|
|
|
|
1051 |
Our current implementation is a layer above Storm block storage |
Our current implementation is a layer above Storm block storage |
1052 |
and indexing. This layer implements a ``load(version-id) -> version`` |
and indexing. This layer implements a ``load(version-id) -> version`` |
1053 |
interface through the following simplified algorithm: |
interface through the following simplified algorithm: |
1120 |
|
|
1121 |
|
|
1122 |
7. Discussion |
7. Discussion |
1123 |
=================================== |
============= |
1124 |
|
|
1125 |
In this section we evaluate Storm's design with regard to previously presented |
In this section we evaluate Storm's design with regard to previously presented |
1126 |
use cases, summarize Storm's implementation status and outline open issues |
use cases, summarize Storm's implementation status and outline open issues |
1127 |
and future work. |
and future work. |
1128 |
|
|
1129 |
|
|
1130 |
7.1 Evaluation |
7.1 Evaluation |
1131 |
-------------- |
-------------- |
1132 |
|
|
1141 |
id returns a location where the data is currently available. If the |
id returns a location where the data is currently available. If the |
1142 |
publisher removes the document permanently, but it is archived elsewhere, |
publisher removes the document permanently, but it is archived elsewhere, |
1143 |
the archives act as peers similarly. When there is no network connection |
the archives act as peers similarly. When there is no network connection |
1144 |
available, but a local copy instead, the lookup points to that. If a |
available, but a local copy instead, Storm can find it. If a |
1145 |
document and a link to it are received independently, e.g. as attachments in |
document and a link to it are received independently, e.g. as attachments in |
1146 |
separate e-mails, or a link to a document in the local intranet is e-mailed, |
separate e-mails, or a link to a document in the local intranet is e-mailed, |
|
(all those links work :o .. this was also discussed in intro already (index)) |
|
1147 |
When people meet live, e.g. on a train, and form an ad-hoc network, they are |
When people meet live, e.g. on a train, and form an ad-hoc network, they are |
1148 |
able to see each other's documents and follow links to them if a |
able to see each other's public documents and follow links to them if a |
1149 |
peer-to-peer implementation of Storm is used (and pools published |
peer-to-peer implementation of Storm is used. |
|
accordingly..? what should be discussed here?). |
|
1150 |
Finally, if a document is split to parts (or content from one copy-pasted to |
Finally, if a document is split to parts (or content from one copy-pasted to |
1151 |
another), links to the elements that are then in the new documents do not |
another), links to the elements that are then in the new documents do not |
1152 |
break, bacause with xanalogical storage the links are actually to spans that |
break, bacause with xanalogical storage the links are actually to spans that |
1153 |
are transcluded in all the documents that show them (as illustrated in |
are transcluded in all the documents that show them (as illustrated in |
1154 |
figure XYZ in section 3, xanalogical storage). |
figure [ref-figdocmovement]_ in section 3). |
1155 |
|
|
1156 |
|
.. (all those links work :o .. this was also discussed in intro already (index)) |
1157 |
|
|
1158 |
*Tracking alternative versions*. Because Storm utilises immutable blocks, |
*Tracking alternative versions*. Because Storm utilises immutable blocks, |
1159 |
each modification to a document creates a new block. When a document is |
each modification to a document creates a new block. When a document is |
1160 |
modified on several independent, unconnected systems, if there are |
modified on several independent, unconnected systems, if there are |
1186 |
7.2 Status, open issues and future work |
7.2 Status, open issues and future work |
1187 |
--------------------------------------- |
--------------------------------------- |
1188 |
|
|
|
|
|
1189 |
[worth to mention, security in general ? -Hermanni] |
[worth to mention, security in general ? -Hermanni] |
1190 |
1) Open issue: We want to be able |
1) Open issue: We want to be able |
1191 |
to copy the blocks from the NYT publication pool to our local computer |
to copy the blocks from the NYT publication pool to our local computer |