13 |
future work in areas such as interoperability, scalability and security is |
future work in areas such as interoperability, scalability and security is |
14 |
identified. |
identified. |
15 |
|
|
16 |
Keywords: data mobility, global unique identifiers, hypermedia management |
Keywords: data mobility, ad-hoc data mobility, mobile hypermedia, |
17 |
system, hyperbase, open hypermedia, structural computing, xanalogical media, |
global unique identifiers, hypermedia management system, hyperbase, |
18 |
permanent storage systems, peer-to-peer networking, distributed hash table, |
open hypermedia, structural computing, xanalogical media, permanent |
19 |
ad-hoc data mobility, mobile hypermedia |
storage systems, peer-to-peer networking, structured overlay networks, |
20 |
|
location inpenpendent routing, distributed hash table |
21 |
|
|
22 |
|
|
23 |
|
|
24 |
1. Introduction |
1. Introduction |
290 |
as regular files, but they are immutable, since any change to the |
as regular files, but they are immutable, since any change to the |
291 |
byte sequence would change the hash (and thus create a different block). |
byte sequence would change the hash (and thus create a different block). |
292 |
Mutable data structures are built on top of the immutable blocks |
Mutable data structures are built on top of the immutable blocks |
293 |
(see Section 6). |
(see Section 6). |
294 |
|
|
295 |
Immutable blocks has several benefits over existing data storing |
Immutable blocks has several benefits over existing data storing |
296 |
techiques: |
techiques: |
301 |
[Previous sentence doesn't parse to me (what level ?) :( -Hermanni] |
[Previous sentence doesn't parse to me (what level ?) :( -Hermanni] |
302 |
To replicate all data from computer A |
To replicate all data from computer A |
303 |
on computer B, it suffices to copy all blocks from A to B that B |
on computer B, it suffices to copy all blocks from A to B that B |
304 |
does not already store. On the other hand, several popular database management |
does not already store. On the other hand for instance, several popular |
305 |
systems (e.g. Lotus Notes [ref]) have complex replication schemes, which may |
database management systems (e.g. Lotus Notes [ref]) have complex |
306 |
led awkward replication conflicts. |
replication schemes, which may led awkward replication conflicts, |
307 |
|
because of they lack the immutable properties of data. |
308 |
[Or does this belong to diff section ? -Hermanni] |
[Or does this belong to diff section ? -Hermanni] |
309 |
|
|
310 |
Storm blocks are MIME messages [ref MIME], i.e., objects with |
Storm blocks are MIME messages [ref MIME], i.e., objects with |
320 |
add(block) |
add(block) |
321 |
delete(block) |
delete(block) |
322 |
|
|
|
[analogy to regular Hash Table/DHT ? -Hermanni] |
|
|
|
|
323 |
Implementations may store blocks in RAM, in individual files, |
Implementations may store blocks in RAM, in individual files, |
324 |
in a Zip archive, in a database or through other means. |
in a Zip archive, in a database or through other means. |
325 |
We have implemented the first three (using hexadecimal |
We have implemented the first three (using hexadecimal |
333 |
be accessible. This makes Storm well suited as a basis |
be accessible. This makes Storm well suited as a basis |
334 |
for implementing experimental projects (such as ours). |
for implementing experimental projects (such as ours). |
335 |
|
|
336 |
When used in a network environment, Storm ids do not provide |
[Example of Microsoft Word ? -Hermanni] |
337 |
a hint as to where in the network a specific block can be found. |
|
338 |
However, current peer-to-peer systems could be used to |
When used in a network environment, Storm IDs do not provide |
339 |
find blocks efficiently in a location independent fashion; for example, |
a hint as to where a specific block is stored in the network. |
340 |
Freenet [ref], recent Gnutella-based clients (e.g. Shareaza [ref]), |
However, many existing peer-to-peer systems could be used to |
341 |
Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers |
find arbitrary blocks in a location independent fashion; |
342 |
|
for example, Freenet [ref], recent Gnutella-based clients |
343 |
|
(e.g. Shareaza [ref]), Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers |
344 |
[e.g. ref: magnet uri]. Footnote:However, we have not put a network |
[e.g. ref: magnet uri]. Footnote:However, we have not put a network |
345 |
implementation into regular use yet and thus can only describe our |
implementation into regular use yet and thus can only describe our |
346 |
design, not report on implementation experience. |
design, not report on implementation experience. |
359 |
[Previous sentence doesn't parse to me: more simple :( -Hermanni] |
[Previous sentence doesn't parse to me: more simple :( -Hermanni] |
360 |
|
|
361 |
Given a peer-to-peer distribution mechanism, it would be possible |
Given a peer-to-peer distribution mechanism, it would be possible |
362 |
to retrieve blocks from any peer online that has a copy |
to retrieve blocks from any participating peer online that has a copy |
363 |
in its cache or permanent storage. This is similar to the Squirrel |
in its local cache or permanent storage. This is similar to the Squirrel |
364 |
web cache [ref] [more refs? -Hermanni], but does not require trust |
web cache [ref] [more refs? -Hermanni], but does not require trust |
365 |
between the peers, since it is possible to check the blocks' integrity by using |
between the peers, since it is possible to check the blocks' integrity by using |
366 |
cryptographic hashes. Since much-requested blocks would be |
cryptographic hashes, as used in many peer-to-peer applications |
367 |
cached on many systems, such a network could deal with XXX |
(e.g. [ref: ed2k/overnet, shareaza]). Since much-requested blocks would be |
368 |
|
cached on many systems, a network could deal with hotspots |
369 |
much more easily. On the other hand, there are privacy |
much more easily. On the other hand, there are privacy |
370 |
concerns with exposing one's browser cache to the outside world. |
concerns with exposing one's browser cache to the outside world. |
371 |
|
|
421 |
in a document does not scale very well. In practice, |
in a document does not scale very well. In practice, |
422 |
characters typed in consecutively are given consecutive ids, |
characters typed in consecutively are given consecutive ids, |
423 |
such as ``...:4``, ``...:5``, ``...:6`` and so on, and |
such as ``...:4``, ``...:5``, ``...:6`` and so on, and |
424 |
operations are on *spans*, consecutive ranges of characters |
operations are on *spans*, i.e. consecutive ranges of characters |
425 |
(``...:4-6``). In Storm, in each editor session we create a |
(``...:4-6``) in a document. In Storm, in each editor session we |
426 |
block with all characters entered in this session (the content type |
create a block with all characters entered in this session (the content type |
427 |
being ``text/plain``). To designate a span of characters |
being ``text/plain``). To designate a span of characters |
428 |
from that session, we use the block's id, the offset of the first |
from that session, we use the block's id, the offset of the first |
429 |
character, and the number of characters in the span. |
character, and the number of characters in the span. |
430 |
This technique was first introduced in [ref ht02 paper]. |
This technique was first introduced in [ref ht02 paper]. |
431 |
|
|
432 |
In Xanadu, characters are stored to append-only *scrolls* |
In Xanadu, characters are stored to append-only *scrolls* |
433 |
when they are typed [ref]. Because of this, we call the blocks |
when they are typed [ref]. Because of this, in Storm, we call the |
434 |
containing the actual characters *scroll blocks*. The documents |
blocks containing the actual characters *scroll blocks*. The documents |
435 |
do not actually contain the characters; instead, they are |
do not actually contain the characters; instead, they are |
436 |
*virtual files* containing span references as described above. |
*virtual files* containing span references as described above. |
437 |
To show a document, the scroll blocks it references are loaded |
To show a document, the scroll blocks it references are loaded |
463 |
systems that allow range queries, such as skip graphs [ref] |
systems that allow range queries, such as skip graphs [ref] |
464 |
and skipnet [ref], may prove useful. |
and skipnet [ref], may prove useful. |
465 |
|
|
466 |
[This might be relevant: |
[This might be relevant also: |
467 |
http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf |
http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf |
468 |
-Hermanni] |
-Hermanni] |
469 |
|
|
485 |
Storm provides a general API for indexing blocks in |
Storm provides a general API for indexing blocks in |
486 |
application-specific ways. We have implemented indexing |
application-specific ways. We have implemented indexing |
487 |
on a local machine, but the interface is designed so that |
on a local machine, but the interface is designed so that |
488 |
implementation on top of a distributed hashtable |
implementation on top of networking overlay (e.g. distributed hashtable) |
489 |
will be trivial. |
will be trivial. |
490 |
|
|
491 |
[Benja, this might be useful for defining Storm APIs for DHTs etc: |
[Benja, this might be useful for defining Storm APIs for DHTs etc: |
497 |
to index blocks provide the following callback |
to index blocks provide the following callback |
498 |
to a Storm pool:: |
to a Storm pool:: |
499 |
|
|
500 |
|
[What is mapping ? We should explain this :) -Hermanni] |
501 |
|
|
502 |
getMappings(block) -> set of (key, value) pairs |
getMappings(block) -> set of (key, value) pairs |
503 |
|
|
504 |
This callback processes a block and returns a set of mappings |
This callback processes a block and returns a set of mappings |
519 |
As an example, the ``getMappings()`` method of our Xanalogical |
As an example, the ``getMappings()`` method of our Xanalogical |
520 |
storage implementation will, for a block containing a document, |
storage implementation will, for a block containing a document, |
521 |
collect all the spans in a document, and return mappings |
collect all the spans in a document, and return mappings |
522 |
from their scroll blocks' ids to the spans and their positions |
from their scroll blocks' IDs to the spans and their positions |
523 |
in the document. When we want to find the transclusions of |
in the document. When we want to find the transclusions of |
524 |
a span, we ``get()`` the mappings for the id of that span's |
a span, we use ``get()`` to get the mappings for the ID of |
525 |
scroll block, and load the document blocks referenced |
that span's scroll block, and load the document blocks referenced |
526 |
by the mappings. |
by the mappings. |
527 |
|
|
528 |
In a networked implementation, each peer is responsible |
In a networked implementation, each peer is responsible |